A stripe threshold segmentation method based on supervised learning

Abstract

This invention, belonging to the field of mechanical processing, proposes a supervised learning-based streak threshold segmentation method. First, laser streak images are acquired and positioned, and a dataset is generated. The training results are used as a classifier. A boundary threshold is determined through a bisection method, and the optimal threshold is evaluated based on the second-order moment, achieving rapid and accurate segmentation of laser streaks from the surrounding background. This method addresses the difficulty of image processing under the influence of various significant and complex background noises, enabling accurate segmentation of laser streaks from the surrounding background in complex industrial environments. Furthermore, by adding a penalty factor to the loss function of a deep neural network, high classification accuracy is achieved for only the C1 type.

Description

Technical Field

[0001] The present invention relates to the field of mechanical processing, and in particular to a stripe threshold segmentation method based on supervised learning. Background Art

[0002] The manufacturing and assembly of large aviation components requires measurement to ensure accuracy, efficiency, and reliability. While current mainstream measurement methods, such as laser trackers, offer high accuracy, they require contact with the object being measured and are extremely inefficient. While maintaining high measurement accuracy, photogrammetry offers a wider range of applications, greater efficiency, and greater convenience, meeting requirements for speed, automation, online portability, and non-contact capabilities.

[0003] Laser streak image processing is a key technology in photogrammetry, and its results directly impact the robustness and accuracy of the system. However, the inability to create an ideal photogrammetric inspection environment for all parts, coupled with the presence of significant and complex background noise in images, makes it difficult to accurately locate and segment targets, presenting significant challenges to laser streak image processing.

[0004] "A method for extracting the center line of laser stripes suitable for dynamic measurement environments" (patent number CN202111494957.2) combines adaptive cropping algorithms, improved gamma correction algorithms, improved variable threshold Otsu threshold segmentation algorithms and other algorithms to overcome the problems of complex background information, uneven brightness distribution of different parts of the light band, and large width differences in single laser stripe images collected in dynamic measurement environments. However, it cannot solve the influence of noise in the image.

[0005] For segmentation purposes, Yinshui He et al. proposed a method based on a saliency visual attention model to locate the contours of laser stripes in the paper "Weld seam profile detection and feature point extraction for multi-pass route planning based on visual attention model" published in 2015. This method uses geometric analysis to locate the position of the stripes in the captured image and segments the stripe contours based on the threshold method and KNN algorithm. The effect is good, but the algorithm is time-consuming and cannot meet the real-time requirements of the processing site.

[0006] However, for photogrammetry of large aviation parts, most existing empirical features are difficult to apply due to the more extreme ratio of laser streak to background area, the greater variety of laser streak shapes, and the greater randomness of the background. Laser streak image processing algorithms still have room for development. Summary of the Invention

[0007] This invention addresses the technical challenge of accurately segmenting laser streaks from their surrounding background in complex industrial environments. By developing a supervised learning-based streak threshold segmentation method, this method achieves rapid and accurate segmentation of laser streaks from their surrounding background by designing a deep neural network architecture, creating a training dataset, and evaluating the optimal threshold.

[0008] The technical solutions of the present invention are as follows:

[0009] A supervised learning-based streak threshold segmentation method acquires and locates light streak images, creates a dataset, trains a deep residual network, uses the training results as a classifier, determines the boundary threshold through bisection, and evaluates the optimal threshold based on the second-order moment to achieve fast and accurate segmentation of laser streaks and surrounding background. The specific steps are as follows:

[0010] The first step is to obtain the light bar image and locate it;

[0011] A laser-assisted binocular vision measurement system was constructed. The system includes two cameras, a turntable 2, a laser emitter 3, and an information acquisition system 7. The turntable 2 is located in the middle of the camera bracket and rotates. The two cameras are located on the camera bracket and on both sides of the turntable 2, namely the left camera 1 and the right camera 4. The laser emitter 3 is located on the turntable 2 and rotates accordingly. The laser emitter 3 emits laser light at the object 5 to form a laser light strip 6. The two cameras are connected to the information acquisition system 7 to obtain the light strip image. The laser light strip area ROL is located based on the multi-feature clustering method.

[0012] The second step is to create a dataset of binary images of laser stripes;

[0013] Since the brightness of laser stripes is not always uniform in the radial direction, especially on abrupt surfaces, multiple thresholds are required for accurate segmentation. To solve this problem, the laser stripe region ROL is evenly divided into multiple images PROL in the vertical direction. The PROL images are segmented based on grayscale and thresholds to obtain training data. The training data is labeled to obtain a dataset of binary laser stripe images.

[0014] All laser stripe binary image categories are:

[0015]

[0016] Among them, l is the image segmentation threshold, rol h,l is the hth image obtained by cutting from the image and segmenting using the image segmentation threshold l. C0 is a white noise image produced by a too small threshold, C1 is a noise-free image produced by an appropriate threshold, and C2 is a black noise image produced by a too large threshold. b0 is the boundary threshold between C0 and C1, and b1 is the boundary threshold between C1 and C2. L is the grayscale level of the image.

[0017] The laser stripe binary image dataset is then randomly divided into training, validation, and test sets for training deep neural networks.

[0018] The third step is to train the deep residual network;

[0019] Set the network structure and parameters to build a deep residual network. The deep residual network includes input layer, maximum pooling layer, Block layer, fully connected layer and loss layer. The network parameters are W = {w, b, γ, β, ω fc};

[0020] The input layer inputs the i-th image PROL at the beginning of the network (i) ;

[0021] The maximum pooling layer, with a step size of 2, is placed below the input layer and above the fully connected layer. The former is used to pool the input data. Downsampling, reducing the number of network parameters and accelerating training, the latter is used to reduce overfitting;

[0022] The formula is as follows:

[0023]

[0024] Among them, 1 is the size of the convolution kernel, are the four thresholds in the 2×2 matrix that performs maximum pooling on the input data;

[0025] The Block layer consists of a convolutional layer, a BatchNorm layer, and a ReLu layer. It is used to perform convolution operations, BatchNorm operations, and linear rectification operations on the output of the previous layer in sequence to obtain more stable training results. The formula is as follows:

[0026]

[0027] Among them, ω and d are the training parameters of the convolution layer, γ and β are the training parameters of the BatchNorm layer, and x is a two-dimensional array of input images. and The input data are The mean and variance of and They are the set after the BatchNorm layer operation and the sequential set of the convolutional layer, BatchNorm layer and ReLu layer;

[0028] The fully connected layer is used to map the feature space calculated by the block layer to the sample label space, reducing the influence of feature position on the classification result. It converts the long vector in the feature space into a short vector and outputs it, aligning it with the label. The formula is as follows:

[0029] g W (PROL (i) )=ip(x H )=ω fc x H (4)

[0030] Among them, x H is the input vector of the fully connected layer, with a length of H = 1024; ω fc is the training parameter of the fully connected layer; g W (PROL (i) ) is the entire network input to PROL (i) The output vector of length 3 corresponds to the three categories C0, C1 and C2 of the laser stripe binary image;

[0031] The loss layer is placed at the output of the deep residual network and is used to calculate the output g of the fully connected layer through the loss function W (x (i) ) and image label y (i) Error, judge the training effect; image label y (i) Defined as:

[0032]

[0033]

[0034] in, is the input image x (i) The k-th dimension label data;

[0035] In order to avoid gradient explosion caused by excessive error and training failure caused by learning rate close to zero, the structure of softmax function and cross entropy error are introduced into the loss function to achieve multi-classification. At the same time, in order to improve the classification accuracy of category C1, the penalty factor is increased to control the error and learning rate.

[0036] The loss function is:

[0037]

[0038] Among them, χ k is a penalty factor that increases the penalty for learning only when the image is misclassified as C1, satisfying χ1>χ0=χ2=1;

[0039] Use the classified dataset in the second step to train the deep residual network;

[0040] The fourth step is to evaluate the optimal threshold based on the second-order moment;

[0041] Use the trained deep residual network to classify the dataset; select an image of type C1; find b0 and b1 through bisection, and the optimal segmentation image threshold is between b0 and b1; extract the shape features of all images between the thresholds b0 and b1; the formula is as follows:

[0042]

[0043] Among them, M h,l is the shape characteristic of the laser stripe of category C1, and are the average values ​​of u and v respectively;

[0044] Then calculate the average value of the second-order moment features of these images; find the relationship between the second-order moment of the image and M h,l The closest binary image That is the optimal segmentation image; its formula is as follows:

[0045]

[0046] in, is the optimal segmentation threshold of the image;

[0047] The optimal segmentation results of all images are stitched together to obtain the segmentation result of the laser stripe image.

[0048] The present invention's beneficial effects: The proposed method accurately segments laser streaks from background noise. By transforming the threshold segmentation task into a classification task, the optimal binary image of the laser streaks is obtained through a trained deep residual network. Furthermore, a supervised learning-based threshold segmentation method is proposed. This method effectively addresses the difficulty of image processing under the influence of significant and complex background noise. Furthermore, by adding a penalty factor to the loss function of the deep neural network, high classification accuracy is achieved for only the C1 type. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the laser-assisted binocular vision measurement system.

[0050] Figure 2 Flowchart of the stripe threshold segmentation method based on supervised learning.

[0051] Figure 3 Schematic diagram of the structure of the deep neural network ResNet for laser stripe binary images.

[0052] In the figure: 1-left camera, 2-turntable, 3-laser transmitter, 4-right camera, 5-object under test, 6-laser light bar, 7-image acquisition system. DETAILED DESCRIPTION

[0053] The specific implementation of the present invention is described in detail below in conjunction with the technical solutions and drawings.

[0054] Figure 1 This is a schematic diagram of a laser-assisted binocular vision measurement system. This embodiment uses two Vieworks CMOS industrial cameras with a resolution of 4096×3072 and a focal length of 35mm, a Lasiris PowerLine series line laser transmitter with a wavelength of 450nm (blue-violet light), a Zhuoli Hanguan electric turntable, and a NI acquisition control system. The line laser transmitter is installed on the turntable 2, and the laser scanning is achieved through the control function of NI, while the binocular camera achieves synchronous acquisition through the acquisition function of NI. In addition, the working distance between the camera and the object is about 1.5m, and the field of view is about 1000×2000mm. The angular velocity of the turntable is 0.1rad / s, and the camera frame rate is set to 20 frames / second. In order to verify the effectiveness of the method proposed in the present invention in locating and segmenting laser stripe images, two groups of experimental images with different laser stripes, different scenes and complex background noises, each containing 100 images, were taken.

[0055] The present invention first obtains laser light stripe images by combining laser with binocular vision and locates them; then, a data set is created using the light stripe images and classified, which is then used to train a deep residual network; finally, the optimal binarization threshold is calculated based on the training results to achieve threshold segmentation of the light stripe, such as Figure 2 shown.

[0056] The specific steps of the method are as follows:

[0057] The first step is to obtain the light bar image and locate it;

[0058] Build a laser-assisted binocular vision measurement system, turn on the auxiliary laser emitter 3 and illuminate the object 5 to be measured. After the left camera 1 and the right camera 4 start collecting data, the laser emitter 3 is rotated by the turntable 2 to make the laser scan the object 5. The auxiliary laser light strip 6 image is collected by the information acquisition system 7 to obtain the light strip image and locate the laser stripe area ROL.

[0059] The second step is to create a dataset of binary images of laser stripes;

[0060] 75 images were taken from the 300 ROL images at equal intervals. The 75 ROL images were evenly cropped into 4 to 12 small images PROL. The pixel size of each PROL was 224×112 to accommodate the input of the deep neural network ResNet. In fact, about 600 PROL images were obtained. Each PROL was then binarized at 25 thresholds at equal intervals to obtain 15,000 binary images. These binary images constitute the ResNet dataset. 60% (9,000 images) of the 15,000 binary image datasets were used as training data, 20% (3,000 images) as validation data, and the remaining 20% ​​(3,000 images) as test data. ResNet was trained based on the training data and validation data.

[0061] The third step is to train the deep residual network

[0062] Establish a deep residual network with a network structure such as Figure 3 As shown;

[0063] The parameter design of each layer of the deep residual network in this embodiment is shown in Table 1;

[0064] Table 1 Structural parameters of each layer in ResNet

[0065]

[0066] Use the classified dataset in the second step to train the deep residual network;

[0067] The fourth step is to evaluate the optimal threshold based on the second-order moment;

[0068] Through training, a trained ResNet is obtained and the ResNet is used as a classifier;

[0069] Considering that if all thresholded binary images were used as ResNet input, ResNet would have to perform H × (L - 1) classifications, which would be very time-consuming. Therefore, a binary search method is used to quickly find b0 and b1, reducing the number of ResNet runs to 2 × log2 [H × (L - 1)]. The pixel size of the ResNet input image is 224 × 112. Therefore, if a 256-grayscale, 2000 × 100 pixel ROL image is divided into 10 200 × 100 pixel PROL images, the number of ResNet classifications is less than 22.

[0070] Considering that the classification results of deep learning have a certain error rate, the binary image with a threshold of b0 or b1 as the output of PROL may be an erroneous result. Although the qualified threshold interval for category 1 is from b0 to b1, images with a threshold near b0 or b1 may be misclassified. Therefore, the shape features of all images between the thresholds b0 and b1 are extracted, and the average of the second-order moment features of these images is calculated as the shape feature M of category C1 (laser stripes). h,l , and then find the image second-order moment and M h,l The closest binary image That is the optimal segmentation image of PROL. Finally, ResNet is used to select the optimal segmentation image of PROL, and then further spliced ​​into the segmentation image of ROL.

[0071] The proposed method can achieve a classification accuracy of 96.4% on the test data. The segmentation method has the highest similarity with the reference segmentation image, and the stability of the segmentation effect is better than other methods, which can better realize the processing of laser streak images.

Claims

1. A stripe threshold segmentation method based on supervised learning, characterized in that: The specific steps are as follows: The first step is to obtain the light bar image and locate it; A laser-assisted binocular vision measurement system is constructed, which includes two cameras, a turntable (2), a laser emitter (3) and an information acquisition system (7); the turntable (2) is located in the middle of a camera bracket and rotates, the two cameras are respectively located on the camera bracket and on both sides of the turntable (2), and the laser emitter (3) is located on the turntable (2) and rotates accordingly; the laser emitter (3) emits laser light to the object to be measured (5) to form a laser light strip (6), the two cameras are connected to the information acquisition system (7) to obtain a light strip image, and the laser light strip area is located based on a multi-feature clustering method; The second step is to create a dataset of binary images of laser stripes; The laser stripe area is evenly divided into multiple images in the vertical direction, and the images are segmented according to grayscale and threshold to obtain training data. The training data is labeled to obtain a dataset of laser stripe binary images. All laser stripe binary image categories are: Among them, l is the image segmentation threshold, rol h,l is the hth image obtained by cutting from the image and segmenting using the image segmentation threshold l, C0 is a white noise image, C1 is a noise-free image, and C2 is a black noise image; b0 is the boundary threshold between C0 and C1, b1 is the boundary threshold between C1 and C2; L is the grayscale level of the image; The laser stripe binary image dataset is then randomly divided into training, validation, and test sets for training deep neural networks. The third step is to train the deep residual network; Set the network structure and parameters to build a deep residual network. The deep residual network includes input layer, maximum pooling layer, Block layer, fully connected layer and loss layer. The network parameters are W = {w, b, γ, β, ω fc }; The input layer inputs the i-th image PROL at the beginning of the network (i) ; A max pooling layer with a stride of 2 is placed below the input layer and above the fully connected layer; The formula is as follows: Among them, 1 is the size of the convolution kernel, are the four thresholds in the 2×2 matrix that performs maximum pooling on the input data; The Block layer consists of a convolutional layer, a BatchNorm layer, and a ReLu layer, which is used to perform convolution operations, BatchNorm operations, and linear rectification operations on the output of the previous layer in sequence. The formula is as follows: Among them, ω and d are the training parameters of the convolution layer, γ and β are the training parameters of the BatchNorm layer, and x is a two-dimensional array of input images. and The input data are The mean and variance of and They are the set after the BatchNorm layer operation and the sequential set of the convolutional layer, BatchNorm layer and ReLu layer; The fully connected layer is used to map the feature space calculated by the block layer to the sample label space, reducing the influence of feature position on the classification result. It converts the long vector in the feature space into a short vector and outputs it, aligning it with the label. The formula is as follows: Among them, x H is the input vector of the fully connected layer, with a length of H = 1024; ω fc is the training parameter of the fully connected layer; g W (PROL (i) ) is the entire network input to PROL (i) The output vector of length 3 corresponds to the three categories C0, C1 and C2 of the laser stripe binary image; The loss layer is placed at the output of the deep residual network and is used to calculate the output g of the fully connected layer through the loss function W (x (i) ) and image label y (i) Error, judge the training effect; image label y (i) Defined as: in, is the input image x (i) The k-th dimension label data; The structure of the softmax function and the cross entropy error are introduced into the loss function to achieve multi-classification. At the same time, in order to improve the classification accuracy of category C1, the penalty factor is increased to control the error and learning rate. The loss function is: Among them, χ k is a penalty factor that increases the penalty for learning only when the image is misclassified as C1, satisfying χ1>χ0=χ2=1; Use the classified dataset in the second step to train the deep residual network; The fourth step is to evaluate the optimal threshold based on the second-order moment; Use the trained deep residual network to classify the dataset; select an image of type C1; find b0 and b1 through bisection, and the optimal segmentation image threshold is between b0 and b1; extract the shape features of all images between the thresholds b0 and b1; the formula is as follows: Among them, M h,l is the shape characteristic of the laser stripe of category C1, and are the average values ​​of u and v respectively; Then calculate the average value of the second-order moment features of these images; find the relationship between the second-order moment of the image and M h,l The closest binary image This is the optimal segmentation image; its formula is as follows: in, is the optimal segmentation threshold of the image; The optimal segmentation results of all images are stitched together to obtain the segmentation result of the laser stripe image.

Images