A line object extraction method based on semantic segmentation

Abstract

The application discloses a line object extraction method based on semantic segmentation. Firstly, a sample image is labeled with a category, a mask image is generated, and a data set is enhanced to obtain a training data set. Then, a semantic segmentation neural network model is constructed, and the model is trained. The trained model is used to infer the image to be detected to obtain a segmentation result. The result obtained by the semantic segmentation is post-processed. The post-processing includes image thinning, cross line recognition and line merging. Finally, the coordinate information of the point formed by the line object in the image is obtained, and the extraction of the line object in the complex image is completed.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a line object extraction method based on semantic segmentation. BACKGROUND

[0002] In industrial production, there are some products or elements in products mainly composed of line objects. Due to the requirement of quality detection, the defects of the products or elements need to be detected. With the development of image processing and computer vision technology, image processing technology has been widely applied in the detection of defects of industrial products.

[0003] In the defect detection using images, an important step is to separate the measured object from the background in the image, that is, image segmentation. Image segmentation is to divide an image into different regions by a certain method, and the same region shows the same or similar gray, color, spatial texture, geometric shape and even semantic features. Image segmentation is an important step between image processing and image analysis, and is also a classic problem in computer vision research.

[0004] In the prior art, the object is still mostly segmented from the image using traditional image segmentation methods, such as common threshold-based segmentation algorithms, edge-based segmentation algorithms and region-based segmentation algorithms, and some other theories such as clustering, mathematical morphology, genetic algorithm and wavelet transform. These traditional algorithms mostly focus on the low-order visual information of the image itself, and are good for images with simple segmentation content, few categories and strong purpose, but cannot utilize the high-level features of the target for segmentation. When facing complex segmentation tasks, the traditional segmentation algorithms cannot achieve the effect for defect detection.

[0005] With the rapid development of deep learning, the technology of convolutional neural network brings new solutions to the field of image processing. Ronneberger et al. proposed U-Net network model based on FCN network (Ronneberger O, Fischer P, Brox T. U-Net: Convolutional Networks for Biomedical Image Segmentation [J]. Springer International Publishing, 2015.), which adds a skip connection between the convolutional part and the symmetrical upsampling part, overcoming the shortcomings of FCN that cannot retain the spatial position information and context information of part of the pixels, resulting in the loss of local features and global features. U-Net is mainly composed of an encoder, a decoder and a skip connection, which reduces the gap between context information and further improves the segmentation performance. Due to the structure similar to U type, it is named UNet network. This network has high segmentation accuracy and is suitable for small sample data.

[0006] The success of U-Net makes many other network structures choose this "U" type network as the backbone of the model. However, the U-Net structure based on the encoding-decoding network also has certain limitations. First, the optimal encoding-decoding depth is unknown in advance, which requires extensive architecture search or inefficient integration of different depth models for testing, that is, different data sets determine the optimal depth of different networks, and the deeper the network, the better the effect is not necessarily; second, the skip connection between the encoder and the decoder imposes unnecessary restrictive fusion scheme, which only forces fusion on the same proportion of feature maps of the encoder and decoder subnets, that is, the same proportion of feature maps from the decoder and the encoder network are not the same in semantics, and there is no reliable theoretical guarantee that they are the best match for feature fusion.

[0007] To solve the above problems of UNet, Zhou et al. proposed UNet++ model based on the improvement of U-Net (Zhou Z, Siddiquee M M R, Tajbakhsh N, et al. UNet++: A Nested U-Net Architecture for Medical Image Segmentation [J]. 2018.), the most important feature of which is to use multiple U-Net networks to stack, which can alleviate the unknown network depth through the effective integration of U-Net with different depths. These U-Net can share an encoder partially, and learn simultaneously through deep supervision. In addition, the UNet++ model redesigns the skip connection, so that the decoder subnetwork can aggregate features with different semantic scales, thus producing a highly flexible feature fusion scheme.

[0008] Although the existing semantic segmentation model can accurately segment the line object from the background, the segmentation result is a binary image, not the coordinate information of the line object. Accurately extracting the coordinates of each point on the line object from the image has become an urgent problem to be solved. SUMMARY

[0009] The present application is exactly aimed at the problem that it is difficult to accurately extract the line object in the complex scene in the prior art, and provides a line object extraction method based on semantic segmentation. First, the sample image is labeled with a class, a mask image is generated, and a data set is enhanced to obtain a training data set. Then, a semantic segmentation neural network model is constructed, and the model is trained. The trained model is used to infer the image to be detected to obtain a segmentation result. The result obtained by semantic segmentation is post-processed, the post-processing includes image thinning, cross line recognition and line merging, and finally the coordinate information of the points composed of the line object in the image is obtained, and the extraction of the line object in the complex image is completed.

[0010] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: a line object extraction method based on semantic segmentation, comprising the following steps:

[0011] S1, constructing a training set sample: using the original picture and the labeled label to generate a mask image through program conversion, forming a data set, and enhancing the data set to form a training data set; the data set enhancement method includes one or more combinations of affine transformation, contrast transformation and elastic deformation;

[0012] S2, constructing a semantic segmentation neural network model: the neural network model mainly includes an encoder, a decoder and a jump connection, the jump connection combines the deep, semantic and coarse-grained feature maps from the decoder subnet with the shallow, low-layer and fine-grained feature maps from the encoder subnet, and realizes fine segmentation;

[0013] S3, model training: training the network model constructed in step S2 according to the training data set obtained in step S1 to obtain a trained network model;

[0014] S4, image segmentation: inputting the sample image to be detected into the trained network model obtained in step S3 for inference segmentation to obtain the segmentation result of the binary mask image;

[0015] S5, image post-processing: post-processing the segmentation result obtained in step S4 to obtain the segmented coordinate information of the image line, and the post-processing at least includes extraction of the line object contour, thinning of the contour, extraction of the line and recognition of the intersection point;

[0016] S6, line merging: performing a set operation on lines whose endpoint distance and endpoint vector direction angle both meet the set threshold, and merging the line parts in the set, to finally obtain the coordinate information of the points in the image formed by each complete line object.

[0017] As an improvement of the application, in the step S1 of constructing the training set sample, a Labelme labeling tool is used to label the sample image in pixel categories to obtain a label data file; the label data file is converted and output to obtain a corresponding mask image; the sample image and the corresponding mask image are simultaneously subjected to horizontal, vertical flipping and rotation transformation, and random brightness, contrast and elastic deformation processing to form a training data set

[0018] As another improvement of the application, the affine transformation in the step S1 of data set enhancement is a linear transformation from two-dimensional coordinates to two-dimensional coordinates, and the sample is enhanced from the whole image, and specifically:

[0019]

[0020] Wherein, (x, y) represents the pixel coordinates on the original image, (x', y') represents the pixel coordinates after transformation, and a, b, c, d, e and f are respectively 6 transformation parameters for controlling translation, rotation, scaling and shearing operations;

[0021] Let Then the formula can also be written as

[0022]

[0023] The matrix A is a 2*2 matrix, which controls the scaling, rotation and shearing linear transformation before and after transformation; wherein, the first column vector And the second column vector Control the scaling and rotation of the x-axis and y-axis before and after transformation; the vector B is a 2*1 column vector, which controls the translation part after transformation; wherein, the first element c controls the translation amount on the x-axis, and the second element f controls the translation amount on the y-axis.

[0024] As another improvement of the application, the contrast transformation in the step S1 of data set enhancement is to enhance the sample from a single pixel, and is specifically represented by the formula x'=αx+β, wherein x' is the gray value of the pixel after transformation, x is the gray value of the pixel before transformation, and a and β respectively determine the contrast and brightness of the transformation.

[0025] As a further improvement of the present application, in the step S1, the elastic deformation in the data set enhancement mode is used to perform sample enhancement from the morphological scale, and is specifically represented by the formula Trans(x+△x(x,y),y+△y(x,y))=I(j,k), wherein, for a point I(j,k) on the original image, a Trans value is obtained after the elastic deformation, wherein a random offset in the interval of (-1, 1) is generated for each pixel point (x, y) of the input image, and △x(x, y) and △y(x, y) are used to represent the displacement of the pixel point in the x and y directions, respectively.

[0026] As a further improvement of the present application, in the step S3, the loss function used in the model training process is: L=β·FL+(1-β)·DL·λ, wherein, FL is a Focal Loss function; DL is a Dice Loss function; β is a hyperparameter used to control the weight relationship between the Focal Loss and the Dice Loss; and λ is a hyperparameter used to balance the order of magnitude imbalance between the Focal Loss and the Dice Loss function values.

[0027] As a further improvement of the present application, the step S5 specifically comprises:

[0028] S51: removing the noise region generated in the step S4 by contour area screening to complete the extraction of the line object contour;

[0029] S52: using the Zhang-Suen algorithm to thin the line object binary image obtained in the step S51 to obtain a line region with a pixel width of 1, and complete the thinning of the contour;

[0030] S53: judging whether a point in the image obtained in the step S51 is a starting point of a line by traversing the neighborhood of each point in the image, and the specific process is as follows: traversing the points in the image, and judging the number of points with pixel values other than 0 in the eight neighborhoods of the point and the point itself: if the number is 0, it means that the region does not contain a line, and is directly skipped; if the number is 1, it means that the region contains an isolated point, and is skipped; if the number is 2, it means that the point is an end point of a line, and is recorded; if the number is 3, it means that the point is in the middle of a line, and is skipped.

[0031] S54: obtaining the coordinates of each point on the line by using a depth-first search algorithm, and judging whether the point belongs to a cross point in the process of the depth-first search to complete the extraction of the line and the identification of the cross point; and the identification of the cross point is specifically as follows: when there are two or more points with pixel gray values of 255 in the eight neighborhoods of the point except the predecessor point of the point, the point is considered as a cross point.

[0032] As a further improvement of the present application, the basic basis for the step S6 to determine whether the lines can be merged is to determine the distance between the two closest breakpoints of the two lines, and to increase the judgment of whether the vector direction angle between the vectors composed of the four endpoints of the two lines meets the set quantitative relationship, which can be expressed as

[0033]

[0034] Wherein, A and B are two endpoints of the same line, C and D are two endpoints of another line, B and C are the two closest endpoints of the two lines, B' and C' are the points of B and C respectively inward n pixels, alpha is Direction angle, beta is Direction angle, gamma is Direction angle, and min_d is the set distance threshold.

[0035] As a further improvement of the present application, the step S6 merges the two line segments into one line, wherein the missing part between the two lines is generated by using the interpolation method to generate the connecting line, and the coordinates of the fitting straight line between the two closest endpoints of the two lines are calculated by the BresenhamLine method based on interpolation, and the three point sets of the two line points and the fitting interpolation point set are merged into one point set according to the order of line growth; the calculation formula of BresenhamLine method is:

[0036]

[0037] Wherein, the calculated results are optimized by rounding.

[0038] Compared with the prior art, the line object extraction method based on semantic segmentation provided by the present application uses a dense skip connection semantic segmentation model to extract line objects, solves the difficult problem that the target cannot be segmented by semantic information in traditional image segmentation, and realizes more accurate segmentation of edges and details compared with UNet network, realizes accurate separation of the detected target, and solves the problem that line objects may cross; the line merging method provided by the present application not only judges the distance between the two line endpoints, but also judges whether the endpoint direction angle meets the condition, greatly reducing the possibility of false merging. The present application provides convenience for further analysis of line objects, and subsequent analysis of line object information including but not limited to line length, curvature parameter at each position, and position relationship between lines can be based on the achievements of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The step flow chart of the line object extraction method based on semantic segmentation of the present application;

[0040] Figure 2 The UNet++ network model structure diagram used for step S2 of the method of the present application;

[0041] Figure 3 The semiconductor X-Ray image original picture 1 provided for embodiment 2 of the present application;

[0042] Figure 4 The segmentation result after using the method for original picture 1 in embodiment 2 of the present application Figure 1 ;

[0043] Figure 5 The semiconductor X-Ray image original picture 2 provided for embodiment 2 of the present application;

[0044] Figure 6 The segmentation result after using the method for original picture 2 in embodiment 2 of the present application Figure 2 ;

[0045] Figure 7 The step flow chart after post-processing of step S5 of the method of the present application;

[0046] Figure 8 The neighborhood numbering diagram defined in the present application;

[0047] Figure 9 The merging operation schematic diagram provided for embodiment 2 of the present application;

[0048] Figure 10 The line merging schematic diagram provided for embodiment 2 of the present application. DETAILED DESCRIPTION

[0049] The present application will be further illustrated below in conjunction with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0050] Embodiment 1

[0051] A line object extraction method based on semantic segmentation, as shown in the figure, includes the following steps: Figure 1

[0052] Step S1, constructing a training set sample: using Labelme annotation tool to annotate the sample image with pixel categories to obtain a label data file; converting the label data file through a program to output a corresponding mask image; performing horizontal, vertical flipping and rotation transformation on the sample image and the corresponding mask image, and performing random brightness, contrast, elastic deformation and other processing to form a training data set.

[0053] ​In the case that the number of original data sets obtained in step S1 is small, data set augmentation is performed using affine transformation, contrast transformation, and elastic deformation and combinations thereof, which can greatly increase the number of effective samples, thereby solving the problem of insufficient training samples due to difficulty in sample acquisition in actual situations. Affine transformation, contrast transformation, and elastic deformation are respectively used for sample augmentation from three aspects of overall image, single pixel, and morphological scale, and the data set augmentation in step S1 uses a combination of the three aspects. Among them, affine transformation can be expressed by the following mathematical expression

[0054]

[0055] wherein I(x,y) represents the pixel coordinates on the original image, I'(x,y) represents the pixel coordinates after transformation, and a,b,c,d,e,f are respectively 6 transformation parameters, which can be used to control translation, rotation, scaling, shearing and other operations. If then the formula can also be written as

[0056]

[0057] The matrix A and the vector B represent the linear part and the translation part of the transformation, respectively. Specifically, the matrix A is a 2*2 matrix, which controls the linear transformation such as scaling, rotation and shearing before and after the transformation. Among them, the first column vector and the second column vector control the scaling and rotation of the x-axis and y-axis before and after the transformation. When the values of a and e are equal, and the values of b and d are zero, it means that only the equal proportion scaling transformation is performed; when the values of a and e are equal, and the values of b and d are not zero, it means that rotation, shearing and scaling transformations are performed. The vector B is a 2*1 column vector, which controls the translation part after the transformation. Among them, the first element c controls the translation amount on the x-axis, and the second element f controls the translation amount on the y-axis. In practical applications, the translation operation of the image can be realized by changing the value of the vector B.

[0058] The contrast transformation can be expressed by the formula x'=αx+β, wherein x' is the gray value of the pixel after transformation, x is the gray value of the pixel before transformation, and α and β respectively determine the contrast and brightness of the transformation.

[0059] Elastic deformation can be expressed by the formula Trans(x+△x(x,y),y+△y(x,y))=I(j,k), wherein the value of Trans is obtained after elastic deformation for the point I(j,k) on the original image, wherein a random offset in the interval (-1,1) is generated for each pixel point (x,y) of the input image, and △x(x,y) and △y(x,y) are used to represent the displacement of the pixel point in the x and y directions

[0060] Step S2, constructing a semantic segmentation neural network model: the neural network model is based on a UNet++ model, mainly including an encoder, a decoder and a skip connection, the skip connection combines deep, semantic, coarse-grained feature maps from the decoder subnet with shallow, low-level, fine-grained feature maps from the encoder subnet, and realizes more fine segmentation of edges and details;

[0061] Step S3, model training: training the network model constructed in step S2 according to the training data set obtained in step S1 to obtain a trained network model;

[0062] The loss function used in step S3 is a weighted mixed loss function of Focal Loss and Dice Loss. Focal Loss is a loss function for handling sample classification imbalance. It focuses on adding weights to the loss corresponding to the sample according to the difficulty of sample recognition, that is, adding a smaller weight α1 to the sample that is easy to distinguish, and adding a larger weight α2 to the sample that is difficult to distinguish, so that the model mainly focuses on difficult samples and increases the weight of difficult samples. Therefore, the expression of Focal Loss loss function can be

[0063] FL(p t )=-α t (1-p t ) γ log(p t )

[0064] The Dice coefficient is a set similarity measure function, which is usually used to calculate the similarity of two samples, which can be represented by , and the Dice Loss is represented as In semantic segmentation, X represents a label image, and Y represents an inference image, so the Dice Loss function can be further represented as

[0065]

[0066] Combining the above Focal Loss and Dice Loss, the final definition of the mixed loss function expression is

[0067] L=β·FL+(1-β)·DL·λ

[0068] Where β is a hyperparameter for controlling the weight relationship between Focal Loss and Dice Loss, and λ is a hyperparameter for balancing the order of magnitude imbalance between Focal Loss and Dice Loss function values. The beneficial effect of this step is that it can effectively solve the problem of inconsistent proportions of difficult and easy samples and unbalanced positive and negative samples in semantic segmentation.

[0069] Step S4, image segmentation: input the image of the sample to be tested into the trained network model obtained in step S3 for inference segmentation to obtain a segmentation result; the segmentation result in this step is a binary mask image, and 0 or 255 of the pixel value thereof represents the category represented by the pixel at the position in the original image. When performing inference, first convert the pth file of Pytorch into the ONNX open neural network exchange format, and use the onnxruntime engine to perform inference.

[0070] Step S5, image post-processing: post-process the segmentation result obtained in step S4 to obtain segmented coordinate information of the image lines; the core of line extraction lies in obtaining ordered coordinate information of points constituting the lines.

[0071] The post-processing of the segmentation result in this step includes extraction of the line object contour, thinning of the contour, extraction of the line, and identification of the intersection point. The findContours function provided by OpenCV is used for extraction of the object contour; the Zhang-Suen thinning algorithm based on iteration is used for thinning of the contour; and the basic method for extraction of the line and identification of the intersection point is the depth-first search algorithm, and in the process of depth-first search, whether there is an intersection point is determined by the number of neighborhood points of each point. Thus, this step specifically includes:

[0072] S51, remove the noise region generated in step S4 by contour area screening to complete extraction of the line object contour;

[0073] S52, thin the line object binary image obtained in step S51 using the Zhang-Suen algorithm to obtain a line region with a pixel width of 1, and complete thinning of the contour;

[0074] S53, determine whether each point in the image obtained in step S51 is a starting point of the line by traversing the neighborhood of each point in the image;

[0075] S54, obtain the coordinates of each point on the line using the depth-first search algorithm, and determine whether the point is an intersection point in the process of depth-first search, to complete extraction of the line and identification of the intersection point.

[0076] The extraction of the line and the identification of the intersection point in the step S5 uses a depth-first search algorithm. Specifically, first, all line endpoints in the image are found, and the eight-neighborhood of the endpoint is judged to determine whether there is a point with a gray value of 255. For convenience of description, let the coordinates of the point P be (x, y), and define the coordinates of the points in the upper, lower, left, and right directions of P as (x, y-1), (x, y+1), (x-1, y), and (x+1, y), respectively, and the coordinates of the points in the upper-left, lower-left, upper-right, and lower-right directions of P as (x-1, y-1), (x-1, y+1), (x+1, y-1), and (x+1, y+1), respectively.

[0077] If one and only one point in the upper, lower, left, and right directions meets the condition, the point is taken as the next point, the original point gray value is set to 0, and the next point is taken as the original point to repeat the above operation; if one and only one point in the upper-left, lower-left, upper-right, and lower-right directions meets the condition, the point is taken as the next point, the original point gray value is set to 0, and the next point is taken as the original point to repeat the above operation; if more than one point in the upper, lower, left, and right directions or in the upper-left, lower-left, upper-right, and lower-right directions meets the condition, it is considered that an intersection point is found, and the above operation is repeated with the points in the neighborhood that meet the condition as the starting points of another line; if no point in the eight-neighborhood meets the condition, it is considered that the end of the line is reached, and the search is terminated. The beneficial effect of this step is that the step S5 separately classifies the starting points of the line segments with the endpoints and intersection points of the lines, and then combines them into complete lines, thereby ensuring that the method is still applicable in the case of line intersection.

[0078] The embodiment uses an improved depth-first search algorithm, which, on the basis of the traditional search algorithm searching for a point meeting the condition, increases the identification of the intersection point and the processing of the intersection point. The line constituting points obtained in the step S54 are arranged in the order of line growth, which facilitates the further analysis and processing of the lines.

[0079] In the step S6, the lines are merged: the lines with the endpoint distance and the endpoint vector direction angle meeting the set threshold value are subjected to a union set operation, and the line parts in the set are merged, and finally the coordinate information of the points constituting each complete line object in the image is obtained.

[0080] The merging of the line in the step S6 is based on the basic idea of the graph theory and the set union, and the merging strategy is improved: as long as the same line segment appears in two sets, the two sets belong to the same connected domain, and they can be merged. Through the optimization and improvement, each line segment can be used multiple times, so that a common line segment can appear in different sets.

[0081] The set union operation and the merging of the line segments in the set are as follows: n line segments are obtained in the step S5, and are denoted as line 1 to line n. Whether two line segments belong to the same connected domain is determined. The basis for determining whether two line segments belong to the same connected domain is whether the vector direction angles of the vectors composed of the four end points of the two line segments meet the assumed quantitative relationship. The line segments meeting the connected relationship are stored in a set. Then, whether there is a common line segment in different sets is determined. If there is a common line segment, the two sets are merged into one set, which indicates that the two sets belong to the same connected domain. After the above operation, m sets are obtained, each of which contains k i line segments, which indicates that there are m disconnected lines in total, and the i-th line is composed of k i sub-line segments. Next, the sets with the sub-line segment number k i greater than 1 are merged, so that the number of line segments in each set is 1. In the merging process, the Bresenham Line algorithm is used, which can give the two-dimensional coordinates of the straight line passing through the two points according to any given two points. The beneficial effect of this step is that each line segment can be used multiple times, so that a common line segment can appear in different sets. Such design is consistent with the actual situation, because in some cases, the refined intersection point is not a single point, but a small common line segment, and the common line segment belongs to different connected domains.

[0082] Further, the basic basis for determining whether the line segments can be merged in the step S6 is to determine the distance between the nearest breakpoints of the two line segments, and to determine whether the vector direction angles of the vectors composed of the four end points of the two line segments meet the assumed quantitative relationship, which can be expressed by the formula

[0083]

[0084] wherein A and B are two end points of the same line, C and D are two end points of another line, B' and C' are points obtained by moving B and C inward by n pixels, respectively, α is the direction angle, β is the direction angle, and γ is the direction angle, min_d is a set distance threshold. The beneficial effect of this step is that not only is it determined whether the closest distance of two line endpoints meets the condition, but under the premise of meeting the closest distance condition, it is assumed that the two closest endpoints are connected, and the direction angle of the vector is used to determine whether such a connection meets the growth trend of the line, which can greatly reduce the misrecognition rate.

[0085] After the above steps, the result obtained by the method provided by the present application is obtained, i.e. the coordinate information of each line in the image

[0086] Example 2

[0087] This embodiment takes the X-ray imaging image of the packaged chip in the semiconductor industry as an example, such as Figure 3 and Figure 5 , Figure 3 and Figure 5 are two original pictures of semiconductor X-rays, and the method of the present application is used to extract the leads in the chip. The purpose of lead extraction is to calculate the length, curvature, position relationship and other parameters of the lead.

[0088] A line object extraction method based on semantic segmentation, the steps of the method comprising:

[0089] Step S1, constructing a training set sample, using the original picture and the labeled label to generate a mask image after conversion, and then using a certain data set enhancement algorithm to form a training data set;

[0090] In order to effectively expand the number of data set samples, affine transformation, contrast transformation and elastic deformation are used for data set enhancement. Affine transformation is a linear transformation between two-dimensional coordinates, which can maintain the "flatness" of two-dimensional graphics, that is, the line common point and point collinear relationship of the original graphics can be maintained unchanged after transformation. Affine transformation can be represented by the following mathematical expression

[0091]

[0092] Wherein, represents the pixel coordinates on the original image, represents the pixel coordinates after transformation, and are 6 transformation parameters, which can be used to control translation, rotation, scaling, shearing and other operations. If then the formula can also be written as

[0093]

[0094] Matrix A and vector B represent the linear part and translation part of the transformation respectively. Specifically, matrix A is a 2*2 matrix which controls the linear transformation such as scaling, rotation and shearing before and after transformation. Among them, the first column vector and the second column vector The scaling and rotation of the x-axis and y-axis before and after the transformation are controlled respectively. When the values of a and e are equal, and the values of b and d are zero, it means that only the equal scaling transformation is performed; when the values of a and e are equal, and the values of b and d are not zero, it means that the rotation, shearing and scaling transformations are performed. The vector B is a 2*1 column vector, which controls the translation part after the transformation. Among them, the first element c controls the translation amount on the x-axis, and the second element f controls the translation amount on the y-axis. In practical application, the translation operation of the image can be realized by changing the value of the vector B.

[0095] The elastic deformation randomly performs affine transformation on the position and value size of each pixel point of the input image. Since the transformation of each pixel point is independent of each other, a certain distortion effect will be generated on the objects of the original image, so it is called elastic deformation. The elastic deformation algorithm used in data set enhancement mainly includes three steps:

[0096] 1) Randomly offset the input image for each pixel point (x, y) in the interval (-1, 1), and use△x(x, y) and△y(x, y) to represent the displacement of the pixel point in the x and y directions respectively;

[0097] 2) Convolve filter the random offset values using a Gaussian kernel knn(0,σ) with standard normal distribution;

[0098] 3) Adjust the deviation using the magnification coefficient α and apply it to the original image.

[0099] △x=G(σ)*(α*Rand(n,m))

[0100] △y=G(σ)*(α*Rand(n,m))

[0101] Trans(x+△x(x,y),y+△y(x,y))=I(j,k)

[0102] For the point I(j, k) on the original image, the Trans value is obtained after elastic deformation. The above formula shows that the change amplitude of elastic deformation is related to α and σ. The larger α is, the more dramatic the deformation is. The larger σ is, the less obvious the deformation is. The smaller σ is, each pixel point will be randomly moved.

[0103] Affine transformation and elastic deformation may produce decimals and thus produce invalid coordinates, so the bilinear interpolation algorithm is used to further calculate the pixel value of the target position. Since the lead in the semiconductor chip itself has a certain deformation ability, i.e. it may produce a certain bending, the new image generated by the elastic deformation method conforms to the actual situation and still has credibility.

[0104] Step S2, constructing a semantic segmentation neural network model, the convolutional neural network mainly includes an encoder, a decoder and a skip connection;

[0105] As shown in Figure 2 The neural network model used in step S2 mainly consists of an encoder, a decoder and a skip connection. The skip connection combines the deep, semantic, coarse-grained feature maps from the decoder subnet with the shallow, low-level, fine-grained feature maps from the encoder subnet, and it is proved that it can effectively restore the fine-grained details of the target object even in a complex background. The redesigned skip connection in the UNet++ network relative to the UNet can aggregate features with different semantic scales on the decoder subnet, thereby producing a highly flexible feature fusion scheme.

[0106] Step S3, training the network model constructed in step S2 according to the training data set in step S1;

[0107] In this step, the loss function used is a weighted mixed loss function of Focal Loss and Dice Loss, and the final definition of the mixed loss function expression is

[0108] L = β·FL + (1-β)·DL·λ

[0109] Where β is a hyperparameter used to control the weight relationship between Focal Loss and Dice Loss, and β = 0.7 is taken here, and λ is a hyperparameter used to balance the order of magnitude imbalance between Focal Loss and Dice Loss function values, and λ = 1 is taken here.

[0110] Step S4, using the trained network model in step S3 to infer and segment the sample image to be tested and obtain the segmentation result;

[0111] Step S5, post-processing the segmentation result in S4, which includes line object contour extraction, contour thinning, line extraction and intersection point recognition, and obtaining the line segmentation coordinate information;

[0112] As shown in Figure 7 The post-processing in step S5 can be divided into the following steps:

[0113] S51, remove the noise area generated in step S4 by contour area screening;

[0114] S52, use the Zhang-Suen algorithm to thin the line object binary image obtained in step S51 to obtain a line region with a pixel width of 1;

[0115] S53, judging whether the point is the start point of the line by traversing the neighborhood of each point in the image obtained in S51, wherein the eight-neighborhood of the point is defined as shown in Figure 8

[0116] S54, obtaining the coordinates of each point on the line by using the depth-first search algorithm, and judging whether the point belongs to the intersection point in the process of the depth-first search;

[0117] Further, the method for judging whether the point is the end point in S53 is to traverse the points in the image, and to judge the number of points whose pixel values are not 0 in the eight-neighborhood of the point and the point itself: if the number is 0, it is indicated that the region does not contain the line, and is directly skipped; if the number is 1, it is indicated that the region contains the isolated point, and is skipped; if the number is 2, it is indicated that the point is the end point of the line, and is recorded; if the number is 3, it is indicated that the point is in the middle of the line, and is skipped.

[0118] Further, the process of the depth-first search algorithm in S54 is shown in Figure 8 The specific process is as follows: taking the coordinates of the end point S obtained in S52, first judging whether the gray value of the point is 255, if the gray value is 0, it is indicated that the point has been visited in the previous operation and is set to 0, and should be directly skipped; if the value of the point is 255, the coordinates of the start point S are assigned to the point P, and the gray values of the points in the eight-neighborhood of the point P are judged. The gray values of the points in the eight-neighborhood exist the following situations:

[0119] (1) in the four directions of up, down, left and right of the point P, that is, the points 2, 7, 5 and 4 in Figure 8 find one and only one point meeting the condition (i.e. the pixel gray value is 255), then the point is selected as the next point N, and the gray value of the original point P is set to 0 to indicate that the point has been visited, and the subsequent operation is not accessed, and the next point N is taken as the original point P to repeat the above operation;

[0120] (2) no point meeting the condition is found in the four directions of up, down, left and right of the point P, but one and only one point meeting the condition is found in the four directions of upper left, lower left, upper right and lower right, that is, the points 1, 3, 6 and 8 in Figure 8 , then the point is selected as the next point N, and the gray value of the original point P is set to 0 to indicate that the point has been visited, and the subsequent operation is not accessed, and the next point N is taken as the original point P to repeat the above operation;

[0121] ​(3) If more than one point meeting the condition is found in the four directions of up, down, left, right or in the four directions of left up, left down, right up, right down, then the point P is considered as a crossing point, the search of the part of the line should be stopped, and the points meeting the condition in the neighborhood are respectively taken as the starting points of another line, and the above operation is repeated;

[0122] (4) If no point meeting the condition is found in the eight-neighborhood, then it is considered that the end of the line is reached, and the search is terminated.

[0123] The method of step S5 has the following advantages compared with the existing line extraction method:

[0124] (1) The order of the points on the extracted line is arranged according to the growth order of the line; if the traversal is performed according to the row-column order, the obtained points are likely not in the growth order of the line, which will bring great trouble to the subsequent further analysis and processing of the line points;

[0125] (2) The crossing points on the line can be recognized; the end points and the crossing points of the line are taken as the separate classification of the line segments, and the subsequent merging into complete lines is ensured, which guarantees that the method can normally function in the case of line crossing.

[0126] Step S6: performing a union set operation on each line meeting the condition, and merging the line parts in the set to finally obtain the coordinate information of each complete line object in the image;

[0127] The union set operation in step S6 is an optimization operation, and its significance is that when the line object is in a complex background and the segmentation result in step S4 has some defects, or the line is crossed, the parts of the line are separated in step S5, and this step is needed to merge the parts of the line into a complete line. It is assumed that after step S5, each line segment numbered 1-12 is obtained from the set. Figure 9 It should be noted that the line segments processed here are all lines with a width of one pixel after the Zhang-Suen thinning algorithm, and in order to facilitate the representation Figure 9 in the set is not a thinned line, it can be assumed that they are all thinned lines. Next, 1-12 twelve points are pairwise judged whether they meet the merging condition, and the judgment basis will be given below. Take 1 and 2 as an example. Figure 9For example, the final result of the line segments that meet the merging condition are stored in the same set, the elements of set A are 1 and 6, the elements of set B are 2 and 7, the elements of set C are 3 and 4, the elements of set D are 4 and 5, the elements of set E are 6 and 11, the elements of set F are 7 and 12, the elements of set G are 8 and 9, and the elements of set H are 9 and 10. However, the obtained set is not the final result, and obviously, the line segment 6 is common to sets A and E, which indicates that sets A and E belong to the same connected domain, and thus sets A and E can be further merged. The same applies to other sets. After the final merging, there are four sets, namely, set A with elements 1, 6 and 11, set B with elements 2, 7 and 12, set C with elements 3, 4 and 5, and set D with elements 8, 9 and 10, which indicates that there are four independent line segments in the graph, and the line segment A can be obtained by merging the line segments 1, 6 and 11 by the Bresenham Line algorithm. The merging method proposed in the method considers that if the same line segment appears in two sets, the two sets belong to the same connected domain and can be merged.

[0128] The method for merging line segment sets in step S6 has the outstanding advantage that each line segment can be reused multiple times, so that a common line segment can appear in different sets. Such a design is in line with the actual situation, because in some cases, the refined intersection point is not a single point, but a small common line segment, and at this time, the common line segment belongs to different connected domains.

[0129] Further, the method for determining whether two line segments meet the merging condition in step S6 has the following specific process: Figure 10 For example, first take two lines, denoted as line 1 and line 2, and set their end points as A, B and C, D respectively. Calculate the distances from A to C, A to D, B to C and B to D. If the smallest distance among the four distances is less than the set threshold min_distance, further verification is performed, otherwise it is directly determined that the condition is not met. After meeting the condition of the previous step, assume that the distances from the end points of line 1 and line 2 to the midpoint A and point C are the smallest, and take the points 20 pixels inward from the end points as A' and C' respectively. Calculate the direction angles of vectors and If the difference between the direction angles of and is less than 120°, and the difference between the direction angles of and is less than 30°, and If the difference between the direction angles is less than 30°, it is considered that the line 1 and the line 2 where the point A and the point C are located originally belong to the same line, and the point A and the point C should be connected, if the above conditions are not met at the same time, it is skipped. Figure 10 For example, through the calculation of the algorithm, it can be judged that the point A and the point C do not meet the merging conditions, and the point D and the point E on the line 2 and the line 3 meet the merging conditions.

[0130] The main innovation point of the step S6 of judging whether the two line segments meet the merging conditions is that not only whether the nearest distance of the two line ends meets the condition is judged, but also under the premise of meeting the nearest distance condition, it is assumed that the two ends with the nearest distance are connected, and the direction angle quantity relationship of the vector is used to judge whether the connection meets the growth trend of the line, which can greatly reduce the misrecognition rate.

[0131] The point coordinate information on each line obtained in the step S6 can be further analyzed and calculated, and the main calculation and judgment contents include whether the position of the line is shifted, whether the length of the line meets the requirements, whether the curvature of each point on the line exceeds the standard, whether the overall bending of the line is qualified, and whether the distance between the lines is qualified. Since the specific coordinates of each point on each line have been obtained in the step S6, and the coordinates are arranged in strict accordance with the growth order of the line, the above calculation items can be obtained by basic algebraic calculation.

[0132] The two original pictures of the embodiment are processed by the method of the present case, and the segmentation result graphs shown in Figure 4 and Figure 6 are obtained, and the above-mentioned embodiment solves the problem of extracting the lead wire from the X-ray imaging image of the semiconductor chip. By using the semantic segmentation model, all the lead wires can be accurately extracted from the image, which breaks through the limitation of the traditional image segmentation that cannot extract the required detection target from the complex background. By using the image thinning and depth-first search method, the coordinate information of each line can be obtained from the binary image of the segmentation result, and the coordinates are arranged in accordance with the growth direction of the line, which solves the problem that some points in the middle of the line may be accessed first when traversing by row or column, which greatly affects the subsequent defect detection related calculation. A line merging method is provided, which provides a post-processing measure when the semantic segmentation result has defects, and the method is also suitable for other scenes where the broken line needs to be reconnected.

[0133] It should be noted that the above content only illustrates the technical idea of the present application, and cannot be used to limit the protection scope of the present application. For ordinary skilled persons in the technical field, some improvements and refinements can be made without departing from the principles of the present application, and these improvements and refinements fall within the protection scope of the claims of the present application.

Claims

1. A line object extraction method based on semantic segmentation, characterized by, The method comprises the following steps: S1, constructing a training set sample: using original pictures and labeled labels to generate mask images through program conversion to form a data set, and enhancing the data set to form a training data set; the data set enhancement method comprises one or more combinations of affine transformation, contrast transformation and elastic deformation; S2, constructing a semantic segmentation neural network model: the neural network model mainly comprises an encoder, a decoder and a jump connection, the jump connection combines deep, semantic and coarse-grained feature maps from the decoder subnet with shallow, low-layer and fine-grained feature maps from the encoder subnet to realize fine segmentation; S3, model training: training the network model constructed in step S2 according to the training data set obtained in step S1 to obtain a trained network model; S4, image segmentation: inputting a sample image to be tested into the trained network model obtained in step S3 for inference segmentation to obtain a segmentation result of a binary mask image; S5, image post-processing: performing post-processing on the segmentation result obtained in step S4 to obtain segmented coordinate information of image lines, and the post-processing at least comprises extraction of line object contours, thinning of the contours, extraction of the lines and identification of intersection points; S6, line merging: performing a union set operation on lines whose endpoint distance and endpoint vector direction angle both meet a set threshold, and merging line parts in the union set to finally obtain coordinate information of points of each complete line object in the image.

2. The line object extraction method based on semantic segmentation according to claim 1, wherein: In the step S1 of constructing a training set sample, a label data file is obtained by using a Labelme labeling tool to label pixel categories of a sample image; the label data file is converted through a program to output a corresponding mask image; the sample image and the corresponding mask image are simultaneously subjected to horizontal and vertical flipping and rotation transformation, and are subjected to random brightness, contrast and elastic deformation processing to form a training data set.

3. The line object extraction method based on semantic segmentation of claim 2, wherein: The affine transformation in the data set enhancement method in the step S1 is a linear transformation from two-dimensional coordinates to two-dimensional coordinates, and sample enhancement is performed from an overall image, and the affine transformation is specifically as follows: Wherein, (x, y) represents pixel coordinates on an original image, (x', y') represents pixel coordinates after transformation, and a, b, c, d, e and f are respectively six transformation parameters for controlling translation, rotation, scaling and shearing operations; Let then the formula can also be written as Matrix A is a 2*2 matrix, which controls the scaling, rotation and shearing linear transformation before and after the transformation; wherein the first column vector and the second column vector respectively control the scaling and rotation of the x-axis and y-axis before and after the transformation; vector B is a 2*1 column vector, which controls the translation part after the transformation; wherein the first element c controls the translation amount on the x-axis, and the second element f controls the translation amount on the y-axis.

4. The line object extraction method based on semantic segmentation of claim 2, wherein: The contrast transformation in the data set enhancement method in the step S1 is sample enhancement from a single pixel, and is specifically represented by a formula x' = αx + β, wherein x' is a pixel transformed gray value, x is a pixel untransformed gray value, and α and β respectively determine the contrast and brightness degrees of the transformation.

5. The line object extraction method based on semantic segmentation of claim 2, wherein: The elastic deformation in the data set enhancement method in the step S1 is sample enhancement from a shape scale, and is specifically represented by a formula Trans(x + △x(x, y), y + △y(x, y)) = I(j, k), wherein a value of Trans is obtained after elastic deformation of a point I(j, k) on an original image, a random offset in an interval of (-1, 1) is generated for each pixel point (x, y) of an input image, and △x(x, y) and △y(x, y) are respectively used to represent a displacement of the pixel point in x and y directions.

6. The line object extraction method based on semantic segmentation according to claim 3 or 4 or 5, characterized in that: In the step S3, the loss function used in the model training process is: L=β·FL+(1-β)·DL·λ, wherein FL is a FocalLoss function; DL is a Dice Loss function; β is a hyperparameter used to control the weight relationship between the Focal Loss and the Dice Loss; and λ is a hyperparameter used to balance the order-of-magnitude imbalance between the Focal Loss and the Dice Loss function values.

7. The line object extraction method based on semantic segmentation of claim 6, wherein: The step S5 specifically includes: S51: removing the noise region generated in the step S4 through contour area screening to complete the extraction of the line object contour; S52: using a Zhang-Suen algorithm to thin the line object binary image obtained in the step S51 to obtain a line region with a pixel width of 1, and complete the thinning of the contour; S53: judging whether each point in the image obtained in the step S51 is a starting point of a line by traversing the neighborhood of the point, specifically: traversing the point in the image, and judging the number of points with pixel values other than 0 in the eight neighborhoods of the point and the point itself: if the number is 0, it is indicated that the region does not contain a line, and the step is directly skipped; if the number is 1, it is indicated that the region contains an isolated point, and the step is skipped; if the number is 2, it is indicated that the point is an end point of a line, and the point is recorded; if the number is 3, it is indicated that the point is in the middle of a line, and the step is skipped; S54: using a depth-first search algorithm to obtain the coordinates of each point on the line, and judging whether the point is a cross point in the process of the depth-first search to complete the extraction of the line and the identification of the cross point; the identification of the cross point specifically includes: when there are two or more points with pixel gray values of 255 in the eight neighborhoods of the point except the predecessor point of the point, the point is considered to be a cross point.

8. The line object extraction method based on semantic segmentation of claim 7, wherein: The basic basis for the step S6 for judging whether the lines can be merged is to judge the distance between the nearest breakpoints of the two lines, and the vector direction angle between the vectors composed of the four end points of the two lines is further judged to determine whether the angle meets a set quantitative relationship, which can be expressed by a formula as follows: where A and B are two end points of one line, C and D are two end points of another line, B' and C' are points which are n pixels inward from B and C respectively, a is the direction angle, β is the direction angle, γ is the direction angle, and min_d is a set distance threshold.

9. The line object extraction method based on semantic segmentation of claim 8, wherein: In the step S6, two line segments are merged into one line, wherein the missing part between the two lines is generated by using an interpolation method to generate a connecting line, and the coordinate points of the fitting straight line between the two end points closest to each other are calculated by using a Bresenham Line method based on interpolation, and the three point sets of the two line point sets and the interpolation point set are merged into one point set in the order of line growth; the calculation formula of the Bresenham Line method is as follows: wherein the calculated result is optimized by rounding. wherein the calculated result is optimized by rounding.

Images