Weld Detection and Segmentation Method and Device Based on Hierarchical Fusion of RGB-D Features

Abstract

The present invention discloses a weld detection and segmentation method based on hierarchical fusion of RGB-D features, comprising the following steps: performing multi-plane segmentation preprocessing on the RGB image and point cloud data of the weld to obtain the RGB-D data of each plane; inputting the obtained RGB-D data of each plane into an improved Mask R-CNN model for recognition and segmentation. The improved Mask R-CNN model includes that in the backbone network, ResNet-FPN extracts features layer by layer from the RGB image and depth image of the RGB-D data respectively, and then layer by layer, the depth features and RGB features are fused using an attention-aware RGB and depth feature fusion module. At each layer where the RGB features are downsampled in the feature pyramid, the corresponding depth features are used for fusion, and the fused features are transmitted layer by layer to the upsampling process of the feature pyramid. The present invention also discloses a corresponding segmentation device. The method and device of the present invention are suitable for the detection of multi-plane welds and can improve the segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to a method and device for weld detection and segmentation, in particular to a method and device for weld detection and segmentation based on hierarchical fusion of RGB-D features. Background Art

[0002] Precisely segmenting and locating welds on multiple planes of a workpiece from a complex industrial scene is an important prerequisite for accurately guiding milling and grinding robots. Traditional geometric methods usually rely on the texture information of welds, are not applicable to relatively complex industrial scenes, and cannot handle the situation where the workpiece is blocked. The 2D segmentation model based on deep learning can solve the problems of noise and unevenness in images. Mask R-CNN is an instance segmentation network, but its mask is transformed from a 28*28 feature map, so the segmentation of the target is not fine enough. The 3D deep neural network for segmentation using point cloud data generally has a slow processing speed.

[0003] Generally speaking, when using deep learning-based methods for weld detection and segmentation, there are the following difficulties and challenges. First, the 3D segmentation model generally has a slow running speed due to the complexity of 3D point clouds, making it difficult to meet the requirements of industrial detection. Second, due to the complex and diverse shapes of welds and the fact that multiple welds on a workpiece may be located on multiple different planes, it is very easy to miss detections just by judging from images. In addition, due to changes in the on-site lighting conditions and the existence of overexposure or underexposure, etc., the classic Mask R-CNN algorithm is prone to problems such as missed detections, incomplete extraction of welds, and low boundary accuracy. Summary of the Invention

[0004] Aiming at the defects of the above-mentioned prior art, the present invention provides a method for weld detection and segmentation based on hierarchical fusion of RGB-D features, and the present invention also provides a device for weld detection and segmentation based on hierarchical fusion of RGB-D features, aiming to improve the accuracy of weld detection and segmentation.

[0005] The technical solution of the present invention is as follows: A weld detection and segmentation method based on hierarchical fusion of RGB-D features, comprising the following steps: Step 1, perform multi-plane segmentation preprocessing on the RGB image and point cloud data of the weld to obtain the RGB-D data of each plane; Step 2, input the obtained RGB-D data of each plane into an improved Mask R-CNN model for recognition and segmentation. The improved Mask R-CNN model includes that in the backbone network, ResNet-FPN extracts features layer by layer from the RGB image and depth image of the RGB-D data respectively, and then layer by layer, the depth features and RGB features are fused using an attention-aware RGB and depth feature fusion module. At each layer where the RGB features are downsampled in the feature pyramid, the corresponding depth features are used for fusion, and the fused features are passed layer by layer into the upsampling process of the feature pyramid.

[0006] Further, the attention-aware RGB and depth feature fusion module includes a channel attention module and a spatial attention module. The channel attention module is the SE module. The attention-aware RGB and depth feature fusion module adds the result of the depth features passing through the SE module, the result of the depth features passing through the spatial attention module and then cross-multiplying with the RGB features, the result of the RGB features passing through the spatial attention module and then cross-multiplying with the features, and the result of the RGB features passing through the SE module for output. The cross-multiplication of RGB features and depth features can better represent the color, texture, shape and depth information of the workpiece image.

[0007] Further, the spatial attention module first compresses the number of channels of the feature map, then extracts features by two dilated convolutions, and finally restores the number of channels and activates the output using the sigmod function. The features extracted by dilated convolution can better retain spatial information.

[0008] Further, the multi-plane segmentation preprocessing of the RGB image and point cloud data of the weld includes the following steps: S1, downsample the point cloud data using voxel filtering; S2, remove the outliers in the point cloud using statistical filtering; S3, use the random sample consensus method to fit a plane to obtain the pitch angle and roll angle, and then use the pitch angle and roll angle to rotate the point cloud of the fitted plane so that the normal vector is parallel to the z-axis; S4, segment the RGB-D image corresponding to the fitted plane according to the correspondence between the RGB image and point cloud data of the workpiece, and remove the fitted plane from the point cloud; S5, repeat steps S3 and S4 to obtain the RGB-D data of each plane.

[0009] Further, in step S5, steps S3 and S4 are repeated until the remaining data in the point cloud is less than a threshold, where the threshold is the percentage of the remaining data volume in the point cloud to the total point cloud data volume.

[0010] Further, the improved Mask R-CNN model includes adding a MaskGIoUHead module to the Mask R-CNN model. The output of the Mask Head of the Mask R-CNN model is concatenated with the input feature map of the Mask Head module after convolution and max pooling to form the input of the MaskGIoU Head module. In the MaskGIoU Head module, 4 convolutional layers and 3 fully connected layers are connected in sequence, and the output of the last fully connected layer is set to the number of classes. The loss function of the MaskGIoU Head module is defined as

[0011]

[0012]

[0013] L MaskGIoU = 1 - GIoU

[0014] where \ is to find the complement set, B is the predicted bounding box, B gt is the ground truth bounding box, and C is the smallest box covering the predicted bounding box and the ground truth bounding box.

[0015] Further, a feature pyramid network is added before the deconvolution layer from the 14×14×256 feature map to the 28×28×256 feature map in the Mask Head module.

[0016] Another technical solution of the present invention is as follows: A weld detection and segmentation device based on hierarchical fusion of RGB-D features, including a control module, the control module includes a processor and a memory, and a computer program is stored on the memory. When the computer program is executed by the processor, the above-mentioned weld detection and segmentation method based on hierarchical fusion of RGB-D features is realized.

[0017] The advantages of the technical solution provided by the present invention are as follows:

[0018] The present invention preprocesses multi-plane segmentation through RGB images and point cloud data and improves the fusion of RGB features and depth features by improving the Mask R-CNN model, thereby improving the detection accuracy of multi-plane welds. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a schematic diagram of the steps of the weld detection and segmentation method based on hierarchical fusion of RGB-D features.

[0020] Figure 2 It is a schematic diagram of the process of the multi-plane segmentation method.

[0021] Figure 3 It is a schematic diagram of the backbone network fusion structure of the improved Mask R-CNN model.

[0022] Figure 4 It is a schematic diagram of the structure of the attention-aware RGB and depth feature fusion module.

[0023] Figure 5 It is a schematic diagram of the network structure of the improved Mask R-CNN model.

[0024] Figure 6 It is the original image of the workpiece to be segmented in the example experiment.

[0025] Figure 7 It is the result image of directly using Mask R-CNN to segment the original image.

[0026] Figure 8 It is the result image of the weld detection and segmentation method based on RGB-D feature hierarchical fusion. Detailed implementation manners

[0027] The following further illustrates the present invention in conjunction with embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. After reading this description, various equivalent modifications made by those skilled in the art to this description all fall within the scope defined by the appended claims of this application.

[0028] Please refer to Figure 1 As shown, the weld detection and segmentation method based on RGB-D feature hierarchical fusion in the embodiment of the present invention includes: Step 1, performing multi-plane segmentation preprocessing on the RGB image and point cloud data of the weld to obtain the RGB-D data of each plane; Step 2, inputting the obtained RGB-D data of each plane into the improved Mask R-CNN model for recognition and segmentation.

[0029] The RGB-D data of the workpiece to be welded at the industrial site is collected by a structured light camera. Among them, the RGB image is stored in the png format as a three-channel grayscale image, and the depth information is stored as an rgb-colored ordered point cloud in the pcd format. The resolution of both is 1920×1200. Different welds may be distributed on the same plane of the workpiece, may also be distributed on different planes, or may even span two different planes, but it is on one plane in the image taken from above it. In order to ensure that a situation that is easily recognized as one weld is recognized as two welds on different planes, it is necessary to first segment the different planes of the workpiece.

[0030] Please refer to Figure 2As shown in the figure, in the multi-plane segmentation method, first, voxel filtering is used to downsample the point cloud. Although random downsampling is more efficient than the voxel filter, it will destroy the microstructure of the point cloud. Using the voxel filter to downsample the point cloud can not only reduce the density of the point cloud but also maintain its structural characteristics, facilitating subsequent plane fitting. The voxel filter uses VoexlGrid, and the input point cloud data is constructed into a three-dimensional voxel grid. After filtering, only the centroid of all points within each voxel is retained. Although using the voxel center is faster, using the centroid as the sampling point can more accurately represent the surface. The specific process of VoexlGrid calculation is to first calculate the side length l of the minimum bounding box of the point cloud x 、l[[ID=③]] y 、l[[ID=⑤]] z ,[[ID=⑦]] [[ID=⑧]]

[0031] [[ID=⑨]] [[ID=⑩]] [[ID=⑪]]

[0032] where x max 、y max 、z max and x min 、y min 、z min are the maximum and minimum values of the point cloud on the three XYZ coordinate axes respectively. Then calculate the number of voxels in the three XYZ directions,

[0033]

[0034] where D x 、D y 、D z represent the number of voxels in the XYZ directions respectively, r is the pre-set side length of the voxel grid, denotes rounding down. Then, calculate the one-dimensional index of each voxel in the point cloud,

[0035]

[0036] where h x 、h y 、h z represent the three-dimensional index of the voxel. Finally, sort the elements in the set h from smallest to largest, retain the centroid of each voxel, and discard other points.

[0037] At this time, there are still some outliers introduced by measurement noise in the point cloud, and their distribution in space is relatively sparse. Statistical filtering is used to remove the outliers in the point cloud. The specific calculation process is as follows: First, calculate the average value μ and standard deviation σ of the distances between each point in the point cloud and other k points. When the average distance of a certain point from k points is within (μ - σ·std, μ + σ·std), it is retained; otherwise, it is judged as an outlier and removed, where k and std are two preset thresholds.

[0038] Then, the random sample consensus (RANSAC) method is used to fit the main plane, which specifically includes the steps: Select three points in the plane as the minimum data set of the model; Use these three points to calculate the plane equation; Calculate the distances from other points to this plane. If the distance is less than the threshold, the point is on this plane; Based on the number of inliers, compare the current plane equation with the previous best plane equation, and retain the plane equation with the largest number of inliers; Stop after iterating N times and return the parameters of the plane equation with the largest number of inliers. The main plane equation of the workpiece point cloud obtained by RANSAC fitting is:

[0039] A·x + B·y + C·z = D

[0040] The main plane point cloud is projected onto the XZ and YZ planes respectively, obtaining two straight-line equations.

[0041]

[0042] According to the parameters of the straight-line equations, the inclination angles of the two straight lines are calculated as the roll angle and pitch angle for rotating the point cloud.

[0043]

[0044] Then, use these two angles to rotate the fitted plane point cloud until the normal vector is parallel to the z-axis.

[0045] Finally, according to the correspondence between the RGB image of the workpiece and the point cloud data, the RGB-D image corresponding to the fitted plane is segmented, the plane in the point cloud is removed, and the remaining outliers are again subjected to RANSAC plane segmentation, that is, repeat the above process until the final remaining number of points in the point cloud is less than the threshold (a certain percentage of the total number), and generally an empirical value of 20% is selected as the threshold.

[0046] To reduce the computational load, the fitting of the plane equation is calculated in the sparse point cloud. The obtained plane equation extracts the point cloud within 4 mm above and below each plane in the original dense ordered point cloud as the point cloud set of the plane. Outliers are filled with NAN values. Finally, the multi-plane dense ordered point cloud is segmented with a resolution of 1920×1200. The depth in the Z-axis direction of the ordered point cloud of each segmented plane is normalized and mapped to [0, 255]. According to the numbers of each point in the width and length of the point cloud, a depth image in the form of a grayscale image is generated. The larger the grayscale value, the deeper the depth. Pure white (i.e., the grayscale value is 255) represents the infinite background. With the ordered point cloud of each plane, corresponding to each pixel point in the RGB image, an RGB image is expanded into multiple images, and only one workpiece plane is retained on each image, and the background is filled with pure white.

[0047] The improved Mask R-CNN model specifically uses ResNet-FPN in the Mask R-CNN model as the backbone network, assuming that the resolutions of the input RGB and depth images are both 1248×768. After ResNet-FPN extracts features from both of them layer by layer, then the depth features and RGB features are layer by layer Figure 3 feature fused using the structure shown. At each layer where the RGB features are downsampled in the feature pyramid, the corresponding depth features are used for fusion, and the fused features are passed layer by layer into the upsampling process of the feature pyramid. The upward arrows in the figure represent the residual blocks (ResidualBlock) of ResNet, fusion is the attention-aware RGB and depth feature fusion module, and the dashed box is the original backbone network ResNet-FPN in the original Mask R-CNN model.

[0048] The attention-aware RGB and depth feature fusion module uses channel attention and spatial attention for both the RGB features and the depth features at the same time. The RGB features and the depth features are cross-fused through spatial attention and finally added to obtain the fused features. The fused features combine the advantages of the RGB features and the depth features and have strong representation ability for the color, texture, shape, and depth information of the image. Please refer to Figure 4 shown, where is matrix addition, is matrix dot multiplication, is the sigmod function, and d-dilated conv is the dilated convolution. The specific structure of the attention-aware RGB and depth feature fusion module is as follows. This fusion module is symmetric. Both the RGB feature and the depth feature pass through a channel attention module and a spatial attention module. The channel attention uses the Squeeze and Excitation (SE) module. For the spatial attention, the number of channels is first compressed to 1 / r of the original, then feature extraction is performed through two dilated convolutions with a convolution kernel of 3×3 and a dilation rate of d. Then the channels are restored and activated using the sigmod function and cross-multiplied with the depth feature (RGB feature). Finally, the four features passing through the attention module are added together to obtain the fused feature, whose size is the same as the original RGB feature and depth feature. The dilated convolution has a larger receptive field, and the features extracted by it can better retain spatial information. The features obtained by cross-multiplying the RGB feature and the depth feature can better represent the color, texture, shape, and depth information of the workpiece image. In this embodiment, the dimensionality reduction coefficient r = 16, and the dilation rate d of the dilated convolution = 4.

[0049] In addition, please combine Figure 5 as shown, where is matrix addition, is matrix concatenation. In this invention, a MaskGIoU Head module is added to the MaskR-CNN model. The output of the Mask Head module is concatenated with the 14×14×256 feature map of the input of the Mask Head module after convolution and max pooling to form the input of the MaskGIoU Head module. In the MaskGIoU Head module, 4 convolutional layers and 3 fully connected layers are used for operations. The outputs of the first two fully connected layers are set to 1024, and the output of the last fully connected layer is set to the number of classes. This model obtains the predicted Mask score in the inference stage, and the predicted Mask score is re-evaluated by multiplying it with the classification score. MaskGIoU Head makes the predicted bounding box of the weld more accurate, can better enclose the entire weld, and improves the localization ability of the weld. In addition, the model of this invention also replaces the IoU used as the IoU of the loss function in MaskIoU Head of Mask Scoring R-CNN with GIoU, effectively solving the problem that it cannot be optimized due to no overlapping part between the Mask predicted bounding box and the true bounding box in the initial stage of training. The loss function of MaskGIoU Head is defined as shown in formula (3),

[0050]

[0051]

[0052] L MaskGIoU= 1 - GIoU (3)

[0053] where \ is the complement set, that is B is the predicted bounding box, B gt is the ground truth bounding box, C is the smallest box covering the predicted bounding box and the ground truth bounding box. For the case where the predicted box and the ground truth box do not overlap, GIoU increases the overlapping area between the two by slowly moving towards the target box to maximize the overlapping area between the predicted box and the ground truth box. This is an iterative process. Using GIoU can solve the problem of gradient disappearance caused by the non - overlapping area between the predicted box and the ground truth box, and can obtain a predicted box with higher accuracy than the IoU loss function.

[0054] To further improve the accuracy of weld seam segmentation, a feature pyramid network is constructed in the Mask Head of the Mask R - CNN model, which fuses multi - scale features of the feature map, making the feature map of 28×28×C in the Mask Head more refined. Since the accurate expression of large weld seam features requires higher requirements for the feature map of 28×28×C, the more refined feature map has a higher - accuracy expression of large weld seam features, and finally improves the boundary quality box of weld seam segmentation when converted into the output weld seam mask. Specifically, a small - scale feature pyramid network is added before the deconvolution layer from the 14×14×256 feature map to the 28×28×256 feature map in the Mask Head, that is, the 14×14×256 feature map is convolved to get 7×7×256 and then deconvolved to get 14×14×256, which is then matrix - added to the 14×14×256 feature map before the aforementioned deconvolution layer, and then passes through the deconvolution layer to get the 28×28×256 feature map. This operation realizes the multi - scale feature fusion of the 14×14×256 feature map, simultaneously focuses on the global features of the weld seam and the details of the weld seam edge part, and improves the segmentation accuracy of large weld seams.

[0055] The weld seam detection and segmentation device based on RGB - D feature hierarchical fusion in the embodiment of the present invention is provided with a control module. The control module can be implemented in a way of hardware, software or the combination of hardware and software, or is a computer device composed of at least one processor and a memory. The memory stores the computer program for implementing the steps of the above - mentioned weld seam detection and segmentation method based on RGB - D feature hierarchical fusion, and the processor is used to execute the computer program on the memory to form the method steps of the above - mentioned embodiment.

[0056] The experimental verification of the method of the present invention is carried out in the following way. The RGB images and point cloud data of the workpiece collected by the area structured light camera are processed to obtain the RGB-D images of each plane of the workpiece. Then, the deep learning image label annotation software labelme is used to perform polygon mask annotation on the RGB-D images, and a corresponding json file is obtained for each RGB-D image. Finally, it is processed into the COCO dataset format and 80% is randomly selected as the training set, and 20% is used as the test set. The training samples are 524 RGB-D images, and the test samples are 132 RGB-D images.

[0057] The annotation type corresponding to the weld instance segmentation task is the target instance. The experiment adopts the COCO dataset format, and the evaluation criteria for weld instance segmentation are shown in Table 1. AP represents the average precision of all classes and is used as the overall standard for the final COCO evaluation. The definition of AP uses the IOU standard. In Table 1, area refers to the number of pixels in the segmentation mask. At the same time, AP also calculates the instance segmentation accuracy of different scale targets, such as large targets, medium targets, and small targets.

[0058] Table 1 Evaluation Index of Weld Dataset

[0059]

[0060] In the improved Mask R-CNN model based on the hierarchical fusion of RGB-D features, the learning rate is set to 0.005. If the initial weights are not set and training starts from scratch, after about 80,000 iterations, the model loss tends to converge. If the initial weights are set to the pre-trained model weights, after about 20,000 iterations, the model loss can tend to converge. After 6 experiments (as shown in Table 2), the performance of different weld instance segmentation methods is compared. The instance segmentation accuracy (AP) of the method in this paper for welds in industrial scenarios reaches 82.017%. An example diagram of the instance segmentation result is as Figures 6 to 8 shown, where Figure 6 is the original image of the workpiece to be segmented. Figure 7 is the result of directly using Mask R-CNN to segment the original image. It misses the weld on the right side of the upper figure, does not completely segment the weld in the middle figure, and identifies one weld as two in the lower figure. Figure 8 is the result of segmentation by the method of the present invention. Among them, #Params is the size of the model parameters, and Fusion is the method of hierarchical fusion of RGB-D data features mentioned above in this paper.

[0061] Table 2 Performance Comparison of Different Weld Segmentation Methods

[0062]

Claims

1. A weld detection and segmentation method based on RGB-D feature layered fusion, characterized in that: The following steps are involved: Step 1: Perform multi-plane segmentation preprocessing on the RGB image and point cloud data of the weld to obtain RGB-D data of each plane; Step 2: Input the obtained RGB-D data of each plane into the improved Mask R-CNN model for identification and segmentation. The improved Mask R-CNN model includes a backbone network in which ResNet-FPN extracts features from the RGB image and depth image of the RGB-D data layer by layer, and then fuses the depth features and RGB features layer by layer using an attention-aware RGB and depth feature fusion module. Each layer in the feature pyramid that downsamples the RGB features is fused with the corresponding depth features, and the fused features are passed layer by layer to the upsampling process of the feature pyramid.

2. The weld detection and segmentation method based on RGB-D feature layered fusion according to claim 1 is characterized in that: The attention-aware RGB and depth feature fusion module includes a channel attention module and a spatial attention module. The channel attention module is an SE module. The attention-aware RGB and depth feature fusion module adds the result of the depth feature passing through the SE module, the result of the cross-point multiplication of the depth feature and the RGB feature after passing through the spatial attention module, the result of the cross-point multiplication of the RGB feature after passing through the spatial attention module and the feature, and the result of the RGB feature passing through the SE module and outputs it.

3. The weld detection and segmentation method based on RGB-D feature layered fusion according to claim 2 is characterized in that: The spatial attention module first compresses the number of channels of the feature map, then extracts features by two dilated convolutions, and finally restores the number of channels and activates the output using the sigmoid function.

4. The weld detection and segmentation method based on RGB-D feature layered fusion according to claim 1 is characterized in that: The multi-plane segmentation preprocessing of the RGB image and point cloud data of the weld includes the following steps: S1, downsampling the point cloud data using voxel filtering; S2, eliminating outliers in the point cloud using statistical filtering; S3, fitting a plane using a random consistency sampling method to obtain a pitch angle and a roll angle, and then using the pitch angle and the roll angle to rotate the point cloud of the fitted plane so that the normal vector is parallel to the z-axis; S4, segmenting the RGB-D image corresponding to the fitted plane according to the correspondence between the RGB image of the workpiece and the point cloud data, and removing the fitted plane from the point cloud; S5, repeating steps S3 and S4 to obtain RGB-D data of each plane.

5. The weld detection and segmentation method based on RGB-D feature layered fusion according to claim 4 is characterized in that: In step S5, steps S3 and S4 are repeated until the amount of remaining data in the point cloud is less than a threshold value, where the threshold value is a percentage of the amount of remaining data in the point cloud to the total amount of point cloud data.

6. The weld detection and segmentation method based on RGB-D feature layered fusion according to claim 1 is characterized in that: The improved Mask R-CNN model includes adding a MaskGIoU Head module to the Mask R-CNN model. The output of the Mask Head of the Mask R-CNN model is concatenated with the input feature map of the Mask Head module after convolution and maximum pooling to form the input of the MaskGIoU Head module. The MaskGIoU Head module uses 4 convolutional layers and 3 fully connected layers to be connected in sequence. The output of the last fully connected layer is set to the number of classes. The loss function of the MaskGIoU Head module is defined as, L MaskGIoU =1-GIoU Where \ is the complement, B is the predicted bounding box, B gt is the true bounding box, and C is the minimum box covering the predicted bounding box and the true bounding box.

7. The weld detection and segmentation method based on RGB-D feature layered fusion according to claim 6 is characterized in that: A feature pyramid network is added before the deconvolution layer from the 14×14×256 feature map to the 28×28×256 feature map in the Mask Head module.

8. A weld detection and segmentation device based on RGB-D feature layered fusion, comprising a control module, the control module comprising a processor and a memory, the memory storing a computer program, characterized in that: When the computer program is executed by the processor, the weld detection and segmentation method based on RGB-D feature layered fusion as described in any one of claims 1 to 7 is implemented.

Images