A weld seam image interactive segmentation method supporting incremental learning

Abstract

The present application relates to the field of workpiece weld detection and segmentation in intelligent grinding, and discloses a weld image interactive segmentation method supporting incremental learning, comprising the following steps: 1) constructing an interactive weld segmentation network based on boundary supervision, including a basic semantic segmentation module, a boundary supervision module and a fusion module; 2) obtaining a batch of weld image samples and accurate labels, and sending the preprocessed samples into the network to obtain an initial segmentation model through training; 3) collecting new weld samples in each actual industrial field, determining the weld through the interactive mode of frame selection and point selection, and obtaining the weld segmentation result by using the current interactive weld segmentation network; 4) sending the new samples and the corrected labels into the network for multiple rounds of incremental training to improve the performance of the segmentation model; and 5) performing trajectory parameterization description on the segmented weld to plan a grinding path for the weld grinding robot. The present application can be used to extract the weld on the surface of a workpiece and guide the robot to mill and grind the weld.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of workpiece weld detection and segmentation in intelligent grinding, and particularly relates to a weld image interactive segmentation method supporting incremental learning. BACKGROUND

[0002] In the post-weld processing task, a visual system is used to complete weld detection and segmentation, thereby guiding an industrial robot to perform post-weld efficient milling and polishing operation, which is an intelligent technical means. Accurate positioning and semantic segmentation of welds on multiple planes of a workpiece from a complex industrial scene are important prerequisites for correctly guiding a milling and polishing robot.

[0003] In actual application, there are many difficulties in collecting weld images. In industrial production, when collecting weld images, some environments such as narrow space, high temperature, dust, vibration and light will affect the quantity, quality and stability of image collection, thereby bringing challenges to the segmentation task of the weld. At the same time, collecting a large number of pixel-level annotations of welds usually consumes huge funds and manpower. In order to ensure the accuracy of model training, the annotator must draw a complex polygon along the weld edge for fine annotation. Therefore, it is difficult to quickly collect new samples for training on site. In order to solve the problem that pixel-level annotation of the weld consumes a large amount of resources, people have proposed interactive image segmentation. The collector can quickly obtain a mask for model training by simple interactive segmentation of the weld image. The main interactive methods include positive and negative clicks, scribbles, extreme point clicks and bounding boxes, etc., which produce a dense binary mask containing the required region. DIOS (see Xu N, Price B, Cohen S, et al. "Deep interactive object selection", Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 373-381.) first introduces deep learning into interactive segmentation, represents positive and negative clicks as a distance map, and connects it with the original image as input, and gives the main pipeline and train / val protocol of interactive segmentation based on clicks.

[0004] Although many interactive segmentation methods have been proposed, interactive segmentation of welds is still a challenging task. Due to the complex actual acquisition site environment, the light is variable, the metal surface of the workpiece to be polished presents different imaging effects under different light conditions, especially the reflection causes the details of the weld to be unstable. In addition, the target proportion of the weld in the image is small, the shape is various, and the shape of the weld image collected in different batches may be completely different. The gray values of the weld and the background environment are close and not easy to distinguish. In addition, due to the strong affinity of aluminum to oxygen, a dense black oxide film is easily formed on the surface of the aluminum alloy at the edge of the weld during the welding process, which will cause part of the actual weld to be misjudged as background or other objects, and will have a certain impact on the accurate segmentation of the weld edge. SUMMARY

[0005] In order to solve the above problems, the present application provides an interactive segmentation method of weld image supporting incremental learning. The method realizes the on-site acquisition of new samples and fast labeling, enables the incremental training mode of improving the performance of the segmentation model by fine-tuning the model, and further improves the performance of the segmentation model.

[0006] In order to achieve the above purpose, the technical scheme of the present application is as follows:

[0007] An interactive segmentation method of weld image supporting incremental learning, comprising the following steps:

[0008] Step 1, step 1, construct an interactive weld segmentation network based on boundary supervision, the input of the network is spliced by the RGB three-channel image of the weld original drawing, a single-channel image containing internal clicks and a single-channel image containing four external clicks. The network structure includes a basic semantic segmentation module, a boundary supervision module and a fusion module;

[0009] Step 2, use a 3D area structured light camera to obtain a workpiece RGB image containing a weld to be polished, accurately label a VOC format data set for training a weld semantic segmentation model, and after a preprocessing step, use it as an initial sample set, input into the network of step S1 for training, to obtain an initial interactive segmentation network model, the preprocessing includes data enhancement, point selection, encoding and splicing.

[0010] Step 3, collect a new batch of a small number of weld samples in each actual deployment of the industrial site, and quickly and roughly label the new samples by rough interaction of clicking outside the weld and selecting an internal point, so as to save the manual accurate labeling of the weld contour. In the process of interactive weld point selection, binary disc coding is used to express the human clicks.

[0011] Step 4, add the new sample collected in step S3 and its labeled mask to the training set, fine-tune the initial interactive segmentation model obtained in step S2, and obtain an optimized interactive segmentation model. After multiple rounds of incremental training, the performance of the interactive segmentation model is continuously improved, and finally the automatic precise segmentation of the weld without any labeling is realized.

[0012] Step 5, the segmented weld is described by trajectory parameterization, the backbone of the weld is extracted, and the coordinate and normal vector of the trajectory point are described. Then the polishing path of the weld is planned, and the weld is milled and polished by the weld polishing robot.

[0013] Specifically, the weld segmentation network in step 1 is composed of an RGB three-channel image of the original image, a single-channel image containing internal clicks, and a single-channel image containing four external clicks. The basic semantic segmentation module in the network takes the improved ResNet and FPN as the backbone network, and integrates the SE self-attention mechanism and the pyramid analysis module. The output of the first conv layer of the semantic segmentation layer backbone network and the output after the third, fourth and fifth conv layers and the SE layer are sent to the boundary supervision module. The edge supervision module adopts three gate structures, focusing on the boundary information of the weld image. Finally, the feature maps of the basic semantic segmentation module and the edge supervision module are fused to generate the final precise segmentation.

[0014] Specifically, the boundary supervision module in step 1 is a GCL layer, which is a combination of attention mechanism and residual structure, allowing features from two branches to interact and ensuring that the edge supervision module only processes boundary-related information.

[0015] Specifically, the target learning loss of the segmentation network in step 1 includes three parts: boundary graph loss Basic semantic segmentation module loss And the final segmentation loss The total loss L is the sum of the three, and its calculation formula is:

[0016]

[0017]

[0018]

[0019] Where λ1, λ2, λ3 are parameters controlling the weights of the three. e represents the boundary graph predicted by the boundary supervision network, represents the standard mask of the weld boundary image, which can be obtained from the standard mask of the weld. And respectively represent the result predicted by the basic semantic segmentation module and the result predicted by the final model after fusion, respectively, y represents the standard mask of the weld image, N represents the number of samples, y i represents the corresponding sample value, i.e. i the true label, is the predicted label of the model for the sample x i , K represents the number of classes in the sample, represents the standard binary cross-entropy loss, represents the standard cross-entropy loss.

[0020] Specifically, the outer point in step 3 is composed of a bounding box containing the weld, and the coordinates of the uppermost point A, the leftmost point B, the lowermost point C and the rightmost point D are found in the weld position coordinates, respectively. The inner point is defined as an internal click located in the center of the weld object to eliminate the ambiguity of the segmentation target, because there may be multiple objects in the same frame. First, the distance map is calculated based on the Euclidean distance transformation, and then the coordinates of the inner point are obtained according to the distance map, which is the position farthest from the distance boundary inside the weld.

[0021] Specifically, the inner and outer points of the click are expressed by binary disc coding in step 3, which solves the problem that the Gaussian coding has weak focus on position information, and can better adapt to the long and narrow shape and small target proportion of the weld.

[0022] Specifically, the incremental training mode in step 4 is a method combining interactive segmentation and manual annotation. Specifically, for the sample segmented by the current model, the sample is sent to the annotation interface and the interactive segmentation result is displayed as a control point on the weld contour, allowing the user to fine-tune the segmentation annotation by dragging the control point on the contour, and then adding the training sample set, thereby greatly shortening the annotation time of the data set mask. With round after round of incremental learning, the model segmentation performance is improved, and the proportion of the number of samples requiring manual fine annotation is less and less.

[0023] The beneficial effects of the present application are:

[0024] 1. A weld interactive segmentation network with boundary supervision is proposed, the SE attention mechanism can adaptively adjust the weight of the feature according to the importance of the feature, improve the discrimination of the feature, make the network pay more attention to the region and feature related to segmentation in the weld image, and let the model automatically pay attention to the channel feature with the largest amount of information, and improve the segmentation performance of the model.

[0025] 2. The boundary supervision module optimizes the information flow between the basic semantic segmentation module and the edge supervision network. The GCL is the core component of the boundary supervision module, which helps the boundary supervision module to process only the shape-related information by filtering out irrelevant information. It can be regarded as a cooperation between the two network modules, in which a more powerful network can form a higher level semantic understanding of the scene to help the other network focus on the relevant part throughout. This cooperation enables the boundary supervision module to effectively process high-resolution images using a shallow architecture.

[0026] 3. In the point selection process of interactive weld segmentation, binary disc coding is used instead of Gaussian coding to express the user's clicks, thereby overcoming the weakness of Gaussian modeling in focusing on position information, and better adapting to long and narrow shapes and small target proportion of welds.

[0027] 4. In actual industrial applications, only through the rough interaction of boxing the weld and selecting an inner point, the fast segmentation and labeling of new samples in the field can be realized. Through the incremental training mode, the segmentation model performance is iteratively improved, and finally the automatic accurate welding without any labeling is realized, reducing the cost of manual labeling. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 Figure 1 is a structural diagram of a weld image segmentation network.

[0029] Figure 2 Figure 2 is a diagram of a robot weld polishing system of the present application.

[0030] Figure 3 Figure 3 is a weld RGB picture collected by a camera.

[0031] Figure 4 Figure 4 is a preprocessed weld image.

[0032] Figure 5 Figure 5 is the overall flow of the method. DETAILED DESCRIPTION

[0033] The present application will be further illustrated below in conjunction with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0034] Step 1, build an interactive weld segmentation network based on boundary supervision, the input of which is a RGB three-channel image of the weld original drawing, a single-channel image containing internal clicks and a single-channel image containing four external clicks. The network structure includes a basic semantic segmentation module, a boundary supervision module and a fusion module.

[0035] Step 2, use a 3D area structured light camera to obtain an RGB image of the workpiece containing the weld to be polished, accurately label it as a VOC format dataset for training the weld semantic segmentation model, and after preprocessing steps as the initial sample set, put it into the network described in step S1 for training, get the initial interactive segmentation network model, the preprocessing includes data enhancement, point selection, encoding, splicing.

[0036] Step 3, collect a new batch of a small number of weld samples in each actual deployment of the industrial site, quickly and roughly label the new samples through the rough interactive method of clicking outside the weld and selecting an inside point, and save the labor of manually accurately labeling the weld contour. Among them, in the interactive weld point selection process, binary disc coding is used to express human clicks.

[0037] Step 4, add the new samples collected in step 3 and their label masks to the training set, fine-tune the initial segmentation model obtained in step 2 to get the optimized interactive segmentation model. After multiple rounds of incremental training to improve the performance of the segmentation model, the final automatic accurate segmentation of the weld without any labeling is realized.

[0038] Step 5, parameterize the trajectory of the segmented weld, extract the backbone of the weld, and parameterize the coordinates and normal vectors of the trajectory points. Then plan the polishing path of the weld, and use the weld polishing robot to perform the milling and polishing task on the weld.

[0039] Specifically, the basic semantic segmentation module described in step 1 is an improved version of the CNN segmentation network, which is based on the ResNet and FPN backbone network, as shown in Figure 1 The last residual block of different Conv features Conv1~Conv5 is denoted as C1, …, C5. Slightly different from FPN, SE attention mechanism is added between C2~C5. The SE attention mechanism can adaptively adjust the weight of the feature according to the importance of the feature, improve the discrimination of the feature, and make the network pay more attention to the region and feature related to segmentation in the weld image. The output feature is denoted as M2, …, M5. In addition, a pyramid parsing module is also added behind the deepest feature C5 layer to improve the understanding of the global context information.

[0040] The boundary supervision module takes the image gradient and the C1, M3, M4 and M5 of the basic semantic segmentation module as input, and generates semantic boundary as output. The network architecture is composed of 3 residual blocks, which are interleaved with gated convolution layers (GCL). The weld boundary semantic segmentation mask can be obtained through the weld semantic segmentation mask, and the boundary supervision module uses binary cross entropy loss to supervise the output of the weld boundary. Specifically, the input features and the gated features are spliced, and after two batch normalization and 1x1 convolution, the result obtained by the Sigmoid function is multiplied by the input features, and then added to the input features, and then the output features are obtained by convolution. In the boundary supervision module, 3 GCLs are used, and they are connected to the third, fourth and last layers of the basic semantic segmentation module, and the feature maps from the basic semantic segmentation module are upsampled using bilinear interpolation.

[0041] Specifically, the target learning loss of the segmentation network in step 1 contains three parts: boundary map loss Basic semantic segmentation module loss And the final segmentation loss The total loss L is the sum of the three, and the calculation formula is:

[0042]

[0043]

[0044]

[0045] Where λ1, λ2, λ3 are parameters that control the weights of the three. e represents the boundary map predicted by the boundary supervision network, represents the standard mask of the weld boundary image, which can be obtained from the standard mask of the weld. And respectively represent the results predicted by the basic semantic segmentation module and the final model after fusion. y represents the standard mask of the weld image. N represents the number of samples, and y i represents the sample value corresponding to each sample x i , i.e. the true label, is the predicted label of the model for the sample x i , K represents the number of classes in the sample, represents the standard binary cross entropy loss, represents the standard cross entropy loss.

[0046] Specifically, the robot polishing system built in step 2 mainly includes a planar structured light camera, a personal computer and a weld polishing robot, as shown in Figure 2 The collected weld images are as shown in Figure 3As shown. Among them, Figure 3 (a) is a normal weld, Figure 3 (b) the edges of (c) (d) form a heat-affected zone under the action of welding thermal cycle, resulting in blurred boundaries, Figure 3 (e) is an incomplete case caused by being blocked by an object, Figure 3 (f) the rightmost boundary is blurred under the double action of backlight and black oxide film, Figure 3 (g) the boundary is blurred due to black oxide film, Figure 3 (h) and (i) part of the edge is difficult to distinguish due to the influence of light.

[0047] Specifically, the preprocessing in step 2 mainly includes the following steps: first, rotate the image, execute according to the probability 0.8, the maximum left rotation angle is 10, and the maximum right rotation angle is 10; Then mirror the image up and down, with a probability of 0.5; Next, mirror the image left and right, with a probability of 0.5; Then scale the image by equal ratio, with a probability of 1, that is, enlarge the entire picture, increase the number of pixels; The last step is to enlarge the image, in order to ensure that the pixels do not change after enlargement, first enlarge the original image by 0.9 times, and then crop according to the original pixel size, with a probability of 0.4. The comparison before and after processing is shown in Figure 4 .

[0048] Specifically, in step 3, the clicks are converted into binary disc encoding and input into the network. Through experiments, it is found that the segmentation effect is more accurate when using binary disc encoding representation.

[0049] Specifically, the incremental training implementation method of the model in step 4 is as follows: first, use the first batch of weld pictures to train the first version of the model Model_v1, and the first data set is denoted as Dataset_v1, and the masks of the data set are obtained by manual annotation. Then, new weld data sets Dataset_v2, Dataset_v3, …, Dataset_vn are obtained in turn, and the weld types in them are not completely similar to the original data set. The samples in the later data sets are first labeled with masks using the interactive segmentation method, and then sent to the labeling interface and displayed as control points on the weld contour with the interactive segmentation result, allowing users to fine-tune the segmentation annotation by dragging the control points on the contour to add the training sample set. Next, let the first version of the model Model_v1 perform incremental training on Dataset_v2, fine-tune the existing model, and obtain the second version of the model Model_v2. Similarly, incremental training is performed on Dataset_v3, …, Dataset_vn to obtain Model_v3, …, Dataset_vn. In this way, the new model obtained by incremental training has a higher degree of adaptation to new data sets than the original model, and finally realizes automatic and accurate segmentation of welds without any labeling.

Claims

1. A method for interactive segmentation of weld images supporting incremental learning, characterized in that, The method comprises the following steps: Step 1, constructing an interactive weld segmentation network based on boundary supervision, the input of the network being a 5-channel image spliced from an RGB three-channel image of a weld original drawing, a single-channel image containing internal clicks and a single-channel image containing four external clicks; the network structure comprising a basic semantic segmentation module, a boundary supervision module and a fusion module; The interactive weld segmentation network based on boundary supervision, the basic semantic segmentation module in the network taking an improved ResNet and FPN as a backbone network, and fusing an SE self-attention mechanism and a pyramid parsing module; the output of the first conv layer of the backbone network after a semantic segmentation layer, and the output after the third, fourth and fifth conv layers and then after an SE layer, being sent into the boundary supervision module together to generate a boundary feature map; the edge supervision module adopting three gate structures to focus on the boundary information of the weld image; finally, the feature maps of the basic semantic segmentation module and the edge supervision module are fused to generate a final accurate segmentation; The boundary supervision module core is a gate structure GCL layer, which is a combined structure combining attention mechanism and residual structure, allowing the feature maps from the two branches to interact, and ensuring that the edge supervision module only processes boundary-related information; The target learning loss of the interactive weld seam segmentation network based on boundary supervision comprises three parts: a boundary map loss , a basic semantic segmentation module loss , and a final segmentation loss , and a total loss is the sum of the three, and the calculation formula is: ； ； ； wherein, , , is a parameter for controlling the weights of the three, e represents the boundary map predicted by the boundary supervision network, represents the standard mask of the weld boundary image, obtained from the weld standard mask; and respectively represent the result predicted by the basic semantic segmentation module and the result predicted by the final model after fusion; y represents the standard mask of the weld image, represents the number of samples, represents the sample value corresponding to each sample , i.e. the true label, is the predicted label of the model for the sample , K represents the number of classes in the sample, represents the standard binary cross entropy loss, represents the standard cross entropy loss; Step 2, acquiring a workpiece RGB image containing a weld to be polished by using a 3D area structured light camera, and accurately labeling a VOC format data set for training a weld semantic segmentation model, the data set being preprocessed as an initial sample set, and being sent into the network of step S1 for training to obtain an initial interactive segmentation network model, the preprocessing including data enhancement, point selection, coding and splicing; Step 3, collecting new weld samples in each actual deployment of an industrial site, and quickly coarsely labeling the new weld samples by an interactive weld point selection method of clicking an external point of the weld and selecting an internal point, wherein a binary disc code is used to express the clicks of a person in the interactive weld point selection process; The external point is composed of a boundary box containing the weld, the coordinates of the uppermost point A, the leftmost point B, the lowermost point C and the rightmost point D in the weld position coordinates are found out respectively, and the internal point is defined as an internal click located in the center of the weld object to eliminate the ambiguity of the segmentation target; first, a distance map is calculated based on Euclidean distance transformation, and then the coordinates of the internal point are obtained according to the distance map, which is the position farthest from the internal boundary of the weld; Step 4, adding the new weld samples collected in step S3 and their coarse labeling masks to the training set, fine-tuning the initial interactive segmentation model obtained in step S2 to obtain an optimized interactive segmentation model, completing one round of incremental training, and thus improving the performance of the interactive segmentation model through multiple rounds of incremental training, and finally realizing automatic accurate segmentation of the weld without any labeling; Step 5, performing trajectory parameterization description on the weld segmented in step S4, extracting the backbone of the weld, and performing coordinate and normal vector parameterization description on the trajectory points of the weld, then planning a polishing path of the weld, and using a weld polishing robot to perform a milling polishing task on the weld.

2. The method of claim 1, wherein: The incremental training mode described in step 4 is a method combining interactive segmentation and manual annotation, specifically, for the sample segmented interactively by the current model, the sample is sent to the annotation interface and the interactive segmentation result is displayed as a control point on the weld contour, allowing the user to fine-tune the segmentation annotation by dragging the control point on the contour, and then adding the training sample set.

Images