Weld joint ROI segmentation method based on active and passive vision fusion

Abstract

The invention discloses a weld ROI segmentation method based on active and passive vision fusion. The method comprises the following steps: synchronously acquiring a two-dimensional image and a three-dimensional point cloud of the surface of a welded workpiece; inputting the trained deep learning network, and outputting a weld joint candidate region and a pixel-level mask graph thereof; establishing a cross-modal mapping relation through geometric calibration, and projecting the three-dimensional point cloud to a two-dimensional image to complete registration and fusion; cutting from the three-dimensional point cloud by using a cross-modal mapping relation to obtain a candidate point cloud region; generating a gray matrix for ray odd-even discrimination according to the pixel-level mask pattern; and each point of the candidate point cloud is projected to a pixel coordinate point by point, ray odd-even discrimination is carried out on the projection points according to the gray matrix, the points conforming to an odd-even rule are determined as welding line points and are aggregated into a welding line ROI point cloud, and the welding line ROI point cloud is output as a segmentation result. According to the method, the adaptive capacity to the unstructured welding seam is effectively improved, the point cloud processing complexity is reduced, and good real-time performance and robustness are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision and automatic welding, and particularly relates to a welding seam ROI segmentation method based on active and passive vision fusion. BACKGROUND

[0002] In the field of automatic welding, automatic polishing and quality detection of welding seams are key links to realize intelligent manufacturing. In traditional methods, operators usually locate and evaluate welding seams through visual observation or simple two-dimensional image processing, but this method is dependent on experience and low in efficiency, and cannot meet the needs of high-precision and large-batch production. With the development of machine vision technology, methods based on three-dimensional point clouds have become an important way to solve this problem because they can provide depth information. By obtaining welding seam point clouds, polishing paths can be planned more accurately and welding seam forming quality can be detected, but the prerequisite is to accurately segment the welding seam region (Region of Interest, ROI). However, three-dimensional point clouds themselves have the characteristics of disorder, sparsity and uneven distribution, which seriously affect the accuracy and efficiency of effective extraction of welding seam features and target positioning.

[0003] To solve this problem, researchers have proposed a variety of methods, for example, based on three-dimensional projection and image binarization to determine the tangent plane to extract welding seam feature points, but this method is highly dependent on plane fitting and is only suitable for regular-shaped welding seams. For complex welding seam point clouds with uneven surfaces, it is difficult to achieve effective segmentation. Some other methods identify and segment welding seam ROI through point cloud simplification filtering, edge recognition, topology restoration and other operations, but these techniques usually rely on regular geometric features and statistical principles of workpieces, so they are only suitable for processing single structured scenes and lack generalization ability in the face of diversified and unstructured environments. In recent years, with the rapid development of deep learning technology, more and more research has begun to use deep learning models for autonomous identification and segmentation of welding seams. However, existing models usually face the problems of long training time and large consumption of computing resources when processing large-scale, high-dimensional point cloud data. At the same time, the generalization ability and robustness of the model still need to be further improved when dealing with complex or irregular welding seam shapes.

[0004] CN202111610858.6 discloses a welding seam detection and segmentation method based on area array structured light 3D vision. The method extracts the contour of the welding seam base material area by combining texture maps and point cloud data, and completes the extraction of the welding seam ROI on this basis. Finally, the welding seam is identified by the SVM support vector machine based on the HOG feature vector. Although this method performs well in terms of reliability and efficiency of welding seam detection, it relies on complex geometric feature extraction and traditional machine learning methods, which limits its generalization ability and robustness when dealing with irregular or varied welding seam shapes. When faced with differences in welding seam surface morphology caused by different materials and welding processes, the stability of feature extraction may be difficult to guarantee. In addition, the method uses texture maps to assist point cloud segmentation, and its performance may be affected in cases where the workpiece surface is reflective or the texture is unclear.

[0005] CN201910140444.8 discloses an automatic teaching method and device for welding seam motion trajectory. The method fuses color images and three-dimensional point cloud images. First, the welding seam area is located in the two-dimensional color image, and then the corresponding local point cloud is extracted in the three-dimensional point cloud through cross-modal mapping. This kind of active and passive vision fusion idea enhances the accuracy of welding seam detection and positioning to a certain extent. However, when locating the welding seam area, this method may mainly rely on two-dimensional image processing, which lacks depth information and is easily affected by lighting, shadows and background complexity, resulting in inaccurate positioning. In addition, during the mapping process from two-dimensional images to three-dimensional point clouds, if the cross-modal registration accuracy is not high, errors may be introduced, affecting the quality of the final welding seam point cloud extraction. These existing technologies have not effectively solved the challenges posed by the inherent characteristics of unordered, sparse and uneven point cloud data to feature extraction and target positioning, especially in complex environments. How to efficiently and accurately segment the subtle welding seam ROI from massive point cloud data remains a technical problem to be solved.

[0006] In summary, the existing technology has problems such as relying on traditional geometric features or machine learning methods, insufficient generalization ability, poor robustness to complex environments, and lack of a stable and reliable method for accurately and efficiently screening welding seam point clouds and forming ROI point clouds in the active and passive vision fusion scheme. The present invention proposes a welding seam ROI segmentation method based on active and passive vision fusion, which combines active structured light three-dimensional scanning with passive two-dimensional image deep learning segmentation, uses cross-modal mapping and ray parity discrimination methods to efficiently and accurately segment welding seam ROI from massive three-dimensional point clouds, solving the problem of low segmentation accuracy and poor robustness in existing technologies when dealing with complex irregular welding seam point clouds. SUMMARY

[0007] The purpose of this part is to outline some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this part and the abstract and title of the specification to avoid obscuring the purpose of this part, the abstract and the title. Such simplifications or omissions cannot be used to limit the scope of the present application.

[0008] In view of the above existing problems, the present application is proposed.

[0009] To solve the above technical problems, the present application provides the following technical solutions: synchronously collecting a two-dimensional image and a three-dimensional point cloud of a welded workpiece surface;

[0010] Input the two-dimensional image into a trained deep learning network to output a weld candidate region and a pixel-level mask image thereof;

[0011] A cross-modal mapping relationship between the three-dimensional point cloud and the two-dimensional image is established through geometric calibration, and the three-dimensional point cloud is projected to the two-dimensional image under a unified coordinate system to complete registration and fusion;

[0012] According to the bounding box of the weld candidate region in the pixel coordinate system, a candidate point cloud region is obtained by cutting from the three-dimensional point cloud using the cross-modal mapping relationship;

[0013] According to the pixel-level mask image, a grayscale matrix for ray parity discrimination is generated;

[0014] In the candidate point cloud region, each point of the candidate point cloud is projected to a pixel coordinate, and the projected points are subjected to ray parity discrimination according to the grayscale matrix. Points that meet the parity rule are determined as weld points and aggregated as a weld ROI point cloud, and the weld ROI point cloud is output as a segmentation result.

[0015] As a preferred scheme of the weld ROI segmentation method based on active and passive visual fusion according to the present application, a deep learning network is used to process the two-dimensional image. The deep learning network is a pre-trained yolov8n-seg deep learning model, and an attention mechanism is introduced to the deep learning network. The deep learning network model includes two parts of target detection and instance segmentation, wherein:

[0016] In the target detection stage, the two-dimensional image weld candidate region detection frame, class, confidence and coordinate information are output;

[0017] In the instance segmentation stage, a weld mask image matching the size of the original image is output.

[0018] As a preferred scheme of the weld ROI segmentation method based on active and passive visual fusion according to the present application, the target detection and instance segmentation specifically include:

[0019] In the target detection stage, the feature maps of three scales N3 (1, 256, 80, 80), N2 (1, 512, 40, 40) and N1 (1, 1024, 20, 20) are input into a decoupled detection head, each feature map is first passed through a 3x3 convolution, and then two 1x1 convolutions are respectively used to branch to predict the bounding box (4 parameters, center coordinates x and y, and size width w and height h) and the branch prediction class probability box (n parameters), finally 8400 candidate boxes are predicted for three scales, each box has (4+n) parameters;

[0020] In the instance segmentation stage, the prototype mask is generated by using convolution upsampling on N2, and the output is (1, 32, 160, 160), and the feature related to classification is obtained from the corresponding (4+n) dimensional vector of each detection box retained in the target detection, and 32 mask coefficients are generated; the 32 mask coefficients of each instance are multiplied by the 32 prototype masks of 160x160 to generate a coarse-grained mask of the instance, and then the region of interest is cropped according to the bounding box coordinates, and the mask is upsampled back to the original image size 1920x1200 to obtain the final instance segmentation result.

[0021] As a preferred scheme of the welding seam ROI segmentation method based on active and passive vision fusion, the attention mechanism is introduced into the deep learning network, which comprises:

[0022] The attention mechanism is composed of a channel attention mechanism and a spatial attention mechanism;

[0023] The mathematical formula of the channel attention mechanism is expressed as:

[0024] M c (F)=σ{MLP[AvgPooL(F)]+MLP[MaxPooL(F)]}

[0025] The mathematical formula of the spatial attention mechanism is expressed as:

[0026] M s (F)=σ{f 7×7 [AvgPooL(F);MaxPooL(F)]}

[0027] Wherein, F is an input feature map, MLP represents a multi-layer perception operation, AvgPool and MaxPool are global maximum pooling and global average pooling operations on F in the channel dimension, sigma represents a Sigmoid activation function, and f 7×7 is a 7x7 convolution.

[0028] As a preferred scheme of the welding seam ROI segmentation method based on active and passive vision fusion, obtaining the cross-modal mapping relationship comprises:

[0029] Based on the two-dimensional image and the three-dimensional point cloud collected synchronously, the cross-modal mapping is performed through the following formula:

[0030]

[0031] Wherein, (X i , Y i , Z i ) is the coordinate of any point p i in the point cloud, β is the coordinate normalization coefficient, M1 and M2 are the camera intrinsic matrix and the extrinsic matrix respectively, (u i , v i ) is the pixel coordinate of p i , and i∈N+ is the number of points.

[0032] As a preferred scheme of the welding seam ROI segmentation method based on active and passive vision fusion, according to the welding seam candidate region, the cross-modal mapping relationship is used to cut the candidate point cloud region from the three-dimensional point cloud, comprising:

[0033] According to the pixel coordinate information of the candidate region, the three-dimensional points falling in the two-dimensional image welding seam candidate region after projection are extracted as the candidate point cloud region.

[0034] As a preferred scheme of the welding seam ROI segmentation method based on active and passive vision fusion, according to the pixel-level mask image, a gray matrix for ray parity discrimination is generated, comprising:

[0035] According to the welding seam candidate region, a pixel gray matrix Lp is constructed, and the length n and the width m of the matrix are determined according to the following formula:

[0036]

[0037] At this time, the gray matrix Lp is represented by the following formula:

[0038]

[0039] Wherein, 「 is the ceiling operator, p x and p y are the pixel point X and Y dimensions respectively, and the element value of the Lp matrix is 255 or 0, wherein 255 is the background gray value, and 0 corresponds to the gray value of the mask contour.

[0040] As a preferred scheme of the welding seam ROI segmentation method based on active and passive vision fusion, the welding seam ROI point cloud is output as the segmentation result, comprising:

[0041] projecting each point of the candidate point cloud to a pixel coordinate based on the cross-modal mapping relationship, emitting a plurality of rays along a predetermined direction outside the image for each projected point, and counting the number of intersection points of each ray with the boundary pixels on the weld mask boundary defined by the grayscale matrix;

[0042] performing point domain inclusion determination according to the parity rule:

[0043] when the number of intersection points is odd or the proportion of the number of rays with odd intersection points is greater than 50%, marking the point as a weld point;

[0044] otherwise, marking the point as a non-weld point;

[0045] aggregating the points marked as weld points into a weld ROI point cloud, and outputting the weld ROI point cloud as the segmentation result.

[0046] As a preferred scheme of the weld ROI segmentation method based on active and passive vision fusion, the candidate point cloud region is traversed and screened, specifically including:

[0047] selecting an optional three-dimensional point (x n ,y n ,z n ) in the candidate point cloud region, and converting it into coordinates (u n ,v n ) according to the cross-modal mapping relationship;

[0048] emitting one ray in the vertical upward direction l and one ray in the horizontal left direction w respectively from the coordinates (u n ,v n ), denoted as l1 and w1;

[0049] counting the number of pixels with value 0 on each ray;

[0050] if the number of pixels with value 0 on the l1 and w1 rays is odd, the three-dimensional point (x n ,y n ,z n ) is a weld point cloud;

[0051] if the proportion of odd and even numbers of pixels with value 0 on the l1 and w1 rays is 50%, a vertical downward ray l2 is introduced, and the number of pixels with value 0 on the l2 ray is calculated, if it is odd, the point is a weld point cloud, otherwise it is a background point cloud;

[0052] without repetition, selecting an optional three-dimensional point in the candidate point cloud region, repeating the above process until the candidate point cloud is traversed, and completing the accurate segmentation of the weld ROI.

[0053] As a preferred scheme of the welding seam ROI segmentation method based on active-passive visual fusion provided by the application, the pre-training data set used is a labeled welding seam picture data set in different forms and scenes, which is divided into a training set, a test set and a validation set, and training is performed according to a 7:2:1 ratio.

[0054] The application has the following beneficial effects: The application realizes efficient, robust and accurate segmentation of the welding seam ROI by cross-modal fusion of active structured light three-dimensional scanning and passive two-dimensional image deep learning segmentation. The technical scheme fully utilizes the advantages of high precision and no influence of light of active vision and the advantages of high efficiency and high robustness of passive vision, realizes rapid coarse positioning of the welding seam area through a deep learning model, and avoids the calculation burden of global processing of massive point clouds.

[0055] Further, the ray parity discrimination method is used to convert the fine segmentation problem of high-dimensional point clouds into pixel-level judgment of low-dimensional images, and fundamentally solves the technical problems of traditional point cloud processing methods in dealing with unordered, sparse and uneven point clouds. Therefore, the application not only improves the accuracy and efficiency of welding seam segmentation, but also significantly enhances the generalization ability and robustness in the face of complex workpiece forms and variable environments. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical scheme of the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0057] Figure 1 The flowchart of the welding seam ROI segmentation method based on active-passive visual fusion shown in the application;

[0058] Figure 2 The example implementation flowchart shown in the application;

[0059] Figure 3 The ray parity discrimination method shown in the application;

[0060] Figure 4 The training effect diagram of Yolov8n-seg and Yolov8n-cbam-seg shown in the application;

[0061] Figure 5 The precision evaluation diagram of Yolov8n-seg and Yolov8n-cbam-seg shown in the application;

[0062] Figure 6The Yolov8n-seg shown in the present application and the position of the introduced attention module are shown in the schematic diagram.

[0063] Figure 7 The attention module structure shown in the present application is shown in the schematic diagram.

[0064] Among them, Figure 7 (a) is the convolution attention module structure, (b) is the spatial attention mechanism, and (c) is the channel attention mechanism. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below in conjunction with the drawings of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments.

[0066] Based on the embodiments in the present application, all other embodiments obtained by the ordinary person in the art without creative labor should belong to the protection scope of the present application.

[0067] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0068] According to the embodiments of the present application, combined with the flow chart shown in Figure 1 The flow chart shown in the present application, a welding seam ROI segmentation method based on active and passive vision fusion, specifically includes the following steps:

[0069] S1, synchronously collecting two-dimensional images and three-dimensional point clouds of the surface of the welded workpiece;

[0070] S2, inputting the two-dimensional image into the trained deep learning network to output the welding seam candidate region and its pixel-level mask image;

[0071] S3, establishing a cross-modal mapping relationship between the three-dimensional point cloud and the two-dimensional image through geometric calibration, projecting the three-dimensional point cloud to the two-dimensional image in a unified coordinate system, and completing registration and fusion;

[0072] S4, according to the bounding box of the welding seam candidate region in the pixel coordinate system, using the cross-modal mapping relationship to cut the candidate point cloud region from the three-dimensional point cloud;

[0073] S5, generating a gray matrix for ray parity discrimination according to the pixel-level mask image;

[0074] S6, projecting each point of the candidate point cloud to a pixel coordinate in the candidate point cloud region, performing ray parity judgment on the projected point according to the gray matrix, judging the point meeting the parity rule as a weld point and aggregating the weld ROI point cloud, and outputting the weld ROI point cloud as the segmentation result.

[0075] It should be noted that by obtaining the three-dimensional spatial geometric information of the welded workpiece, i.e., three-dimensional point cloud data, a high-quality spatial reference is provided for active-passive vision fusion. The active structured light scanning is not affected by the ambient light, and the grating pattern can be actively projected. Even in the case of weld surface reflection or unclear texture, the point cloud can be stably and accurately obtained, providing reliable raw data for subsequent accurate segmentation. By obtaining the two-dimensional texture and color information of the welded workpiece, the three-dimensional point cloud geometric information is complementary. The two-dimensional image contains rich pixel-level details and visual features, providing high-dimensional feature input for the subsequent deep learning model. By using the two-dimensional image, large-scale weld area rough positioning can be efficiently performed, thereby greatly reducing the range of subsequent three-dimensional point cloud processing and significantly improving the overall algorithm efficiency.

[0076] Further, by using the deep learning model, fast rough positioning and pixel-level segmentation of the weld are realized, which can effectively cope with the challenges of complex background, light changes and diverse weld shapes, avoid complex feature engineering and rule setting in traditional methods, and greatly improve the robustness and efficiency of weld detection.

[0077] It should be further noted that the accurate correspondence between the three-dimensional point cloud and the two-dimensional image is established, and the data of the two modalities are fused in a unified coordinate system. The advantages of data of different dimensions and types are combined. Through accurate registration, the rich pixel features in the two-dimensional image can be used to guide the segmentation of the three-dimensional point cloud, and the semantic information in the three-dimensional space is mapped to the two-dimensional space, greatly enhancing the expression ability of the weld features and laying a solid foundation for subsequent accurate segmentation.

[0078] Preferably, the output result of the deep learning network is used to guide the cropping and screening of the three-dimensional point cloud, i.e., a small range of candidate point cloud region is first cropped in the three-dimensional point cloud through the two-dimensional bounding box, avoiding global processing of massive point clouds; then the pixel-level mask image is converted into a gray matrix for ray parity judgment, providing a basis for subsequent accurate judgment, realizing an effective transition from rough positioning to fine segmentation, greatly reducing the amount of calculation by limiting the processing range, significantly improving the efficiency of subsequent point cloud processing, and ensuring that only the area most likely to have a weld is subjected to fine operation.

[0079] Preferably, the ray parity discrimination method is used to accurately screen the points in the three-dimensional point cloud under the guidance of the two-dimensional pixel-level mask map. After each point is projected onto a two-dimensional image, the parity of the pixel position in the gray matrix can be used to determine whether it is located inside the weld. This method effectively avoids the complex geometric calculation and topological analysis commonly used in three-dimensional point cloud segmentation. By converting the three-dimensional point cloud segmentation problem into a pixel discrimination problem on a two-dimensional image, the high precision of the pixel-level mask map is utilized to ensure that the obtained weld ROI point cloud is of high purity and high integrity.

[0080] The implementation process and / or effects of some examples of the present application will be described in more detail below in conjunction with the schematic diagrams shown in the drawings and some preferred or optional examples of the present application. Figures 2-7

[0081] A binocular active line structured light camera is used to scan the welded workpiece and synchronously acquire two-dimensional images and three-dimensional point cloud data of the weld area. A CBAM is introduced into the yolov8n-seg, and a labeled weld picture dataset is used for training. In this example, the training set includes 1350 pictures, the verification set includes 270 pictures, and the test set includes 180 pictures. The initial learning rate is 0.01, the weight decay coefficient is 0.0005, and the training effect is as shown in Figure 4 In the figure, box_loss represents the bounding box loss, seg_loss represents the segmentation loss, cls_loss represents the classification loss, and dfl_loss represents the distribution focal loss.

[0082] The two-dimensional image is input into the trained deep learning network model, which includes two parts of target detection and instance segmentation. The target detection part outputs the center point coordinates (x, y) and the size (width w, height h) of the weld candidate region, and the instance segmentation part outputs the weld pixel-level mask map.

[0083] It should be noted that the dataset used for pre-training is a labeled weld picture dataset in different forms and scenes, which is divided into a training set, a test set, and a verification set according to a 7:2:1 ratio.

[0084] Specifically, the two-dimensional image is processed using a deep learning network, which is a pre-trained yolov8n-seg deep learning model. An attention mechanism is introduced into the deep learning network model, which includes two parts of target detection and instance segmentation. Among them:

[0085] In the target detection stage, the two-dimensional image weld candidate region detection frame, class, confidence and coordinate information are outputted;

[0086] In the instance segmentation stage, the weld mask map matching the size of the original image is outputted.​

[0087] It should be noted that the target detection and instance segmentation specifically include:

[0088] In the target detection stage, the feature maps of three scales N3(1, 256, 80, 80), N2(1, 512, 40, 40), and N1(1, 1024, 20, 20) are input into the decoupled detection head. Each feature map is first passed through a 3x3 convolution, and then two 1x1 convolutions are performed to branch the prediction bounding box (4 parameters, center coordinates x, y, and size width w, height h) and branch the prediction class probability box (n parameters). Finally, 8400 candidate boxes are predicted for each scale, and each box has (4+n) parameters.

[0089] In the instance segmentation stage, convolution upsampling is used to generate a prototype mask for N2, outputting (1, 32, 160, 160). For each detection box retained by the target detection, the features related to classification are obtained from the corresponding (4+n) dimensional vector to generate 32 mask coefficients. The 32 mask coefficients of each instance are multiplied by the 32 prototype masks of 160x160 to generate a coarse-grained mask for the instance. Subsequently, the region of interest is cropped according to the bounding box coordinates, and the mask is upsampled back to the original image size of 1920x1200 to obtain the final instance segmentation result.

[0090] As an example, (1) first scale the two-dimensional picture to the model standard size 640x640, and normalize the pixels, that is, divide the pixel value from 0-255 by 255, scale to 0-1 range, and then adjust the channel order to the format of (batch, channel, high, wide), that is, (1, 3, 640, 640); (2) perform downsampling, convolution kernel 6x6, step = 2, padding = 2 convolution, output channel number 64, output size (1, 64, 320, 320); (3) perform convolution kernel = 3x3, step = 2, padding = 1 convolution, output size (1, 128, 160, 160), and then pass through the C2f module to perform feature extraction, and the output size is still (1, 128, 160, 160); (4) introduce the first CBAM attention module, the input size is (1, 128, 160, 160), the channel attention submodule weights the channel dimension C = 128, and the spatial attention submodule weights the spatial dimension 160x160, focusing on the key area, and the output size remains unchanged, being (1, 128, 160, 160); (5) then recycle steps (3), (4) twice, and then loop step (3) once, in the 3 times of convolution, convolution kernel = 3x3, step = 2, padding = 1, and the output size is P3(1, 256, 80, 80), P4(1, 512, 40, 40), (1, 1024, 20, 20) in turn; (6) pass through the SPPF (Spatial Pyramid Pooling Fast) layer, perform parallel multiple maximum pooling layers (kernel = 5x5, step = 1, padding = 2), output P5(1, 1024, 20, 20); perform feature fusion upsampling: upsample P5 by 2 times, splice with P4 channel, then reduce the channel number through the c2f structure to fuse the features, output N4(1, 512, 40, 40), then upsample N4 by 2, and then splice with P3 in the channel dimension, and then reduce the channel number through the c2f structure to fuse the features, output N3(1, 256, 80, 80); (7) perform feature fusion downsampling: downsample N3 through CBS (convolution kernel = 3x3, step = 2, padding = 1), then splice with N4 in the channel dimension, then fuse the features through the c2f structure, output N2(1, 512, 40, 40), continue to downsample N2 through CBS, splice with P5 in the channel dimension, and then fuse the features through the c2f structure, output N1(1, 1024, 20, 20).

[0091] By way of example, the attention mechanism is introduced into the deep learning network, which includes:

[0092] The attention mechanism is composed of a channel attention mechanism and a spatial attention mechanism;

[0093] The mathematical formula of the channel attention mechanism is expressed as:

[0094] M c(F) = σ{MLP[AvgPooL(F)] + MLP[MaxPooL(F)]}

[0095] The mathematical formula of the spatial attention mechanism is expressed as:

[0096] M s (F) = σ{f 7×7 [AvgPooL(F); MaxPooL(F)]}

[0097] Where F is the input feature map, MLP represents the multi-layer perception operation, AvgPooL and MaxPool are the channel dimension global maximum pooling and global average pooling operations on F, sigma represents the Sigmoid activation function, and f 7×7 is a 7x7 convolution.

[0098] Based on the two-dimensional image and the three-dimensional point cloud collected synchronously in S1, the cross-modal mapping is performed by the following formula:

[0099]

[0100] Where (X i , Y i , Z i ) is the coordinate of any point p i in the point cloud, beta is the coordinate normalization coefficient, M1 and M2 are the camera intrinsic matrix and the extrinsic matrix respectively, (u i , v i ) is the pixel coordinate of p i , and i is the number of points.

[0101] The cross-modal mapping relationship between the three-dimensional point cloud and the two-dimensional image is established through the geometric calibration of S3, the three-dimensional point cloud is projected point by point to the two-dimensional image, and the registration and fusion are completed in the unified coordinate system.

[0102] According to the bounding box of the weld candidate region in the pixel coordinate system, the candidate point cloud region is obtained by cutting from the three-dimensional point cloud using the cross-modal mapping relationship, and the specific steps are as follows:

[0103] Project any point (X i , Y i , Z i ) in the point cloud to the picture pixel coordinate (u i , v i ), when the following two formulas are satisfied simultaneously, the point cloud is the candidate point cloud set:

[0104]

[0105] Where x, y, w, h are the center point coordinate (x, y) and size (width w, height h) obtained in S3.

[0106] According to the above formula, the weld point cloud candidate region can be cut out, that is, according to the pixel coordinate information of the candidate region, the three-dimensional points falling in the two-dimensional image weld candidate region after projection are extracted as the candidate point cloud region.

[0107] It should be noted that the gray matrix for ray parity discrimination generated according to the pixel-level mask map includes:

[0108] According to the weld candidate region, a pixel gray matrix Lp is constructed, and the length n and the width m of the matrix are determined according to the following formula:

[0109]

[0110] At this time, the gray matrix Lp is represented by the following formula:

[0111]

[0112] Wherein, " is a rounding up operator, p x and p y are the pixel point X and Y dimensions respectively, and the element value of the Lp matrix is 255 or 0, wherein 255 is the background gray value, and 0 corresponds to the gray value of the mask contour.

[0113] Further, in the candidate point cloud region, each point of the candidate point cloud is projected to the pixel coordinate, the ray parity discrimination is performed according to the gray matrix, the points meeting the parity rule are determined as the weld points and aggregated as the weld ROI point cloud, and the weld ROI point cloud is output as the segmentation result.

[0114] In the preferred embodiment, the weld ROI point cloud is output as the segmentation result, including:

[0115] Based on the cross-modal mapping relationship, each point of the candidate point cloud is projected to the pixel coordinate, a plurality of rays are emitted along a predetermined direction to the outside of the image for each projected point, and the intersection points of each ray and the boundary pixels on the weld mask boundary defined by the gray matrix are counted;

[0116] According to the parity rule, the point domain inclusion is determined:

[0117] When the intersection point number is odd or the proportion of the number of rays with odd intersection points is greater than 50%, the point is marked as a weld point;

[0118] Otherwise, the point is marked as a non-weld point;

[0119] The points marked as weld points are aggregated as the weld ROI point cloud, and the weld ROI point cloud is output as the segmentation result.

[0120] As an example, the candidate point cloud region is traversed and screened, specifically including:

[0121] In the candidate point cloud region, optionally a three-dimensional point (x n ,y n ,z n ) is converted into coordinates (u n ,v n ) according to the cross-modal mapping relationship;

[0122] Starting from the coordinates (u n ,v n ), respectively, one ray is emitted in the vertical upward direction l and the horizontal left direction w, denoted as l1 and w1.

[0123] The number of pixels with a value of 0 on each ray is counted.

[0124] If the number of pixels with a value of 0 on the l1 and w1 rays is odd, the three-dimensional point (x n ,y n ,z n ) is a weld point cloud.

[0125] If the number of pixels with a value of 0 on the l1 and w1 rays is odd and even, each accounts for 50%, a vertically downward ray l2 is introduced, and the number of pixels with a value of 0 on the l2 ray is calculated. If it is odd, the point is a weld point cloud, otherwise it is a background point cloud.

[0126] Without repetition, one three-dimensional point is selected in the candidate point cloud region, and the above process is repeated until the candidate point cloud is traversed, and the accurate segmentation of the weld ROI is completed.

[0127] Referring to Figure 7 , the working principle of its convolution attention module is as follows: the feature input passes through the channel attention mechanism, respectively performs global maximum pooling (MaxPool) and average pooling (AvgPool) on each channel, then enters the shared fully connected layer (Share MLP) and the activation function (such as ReLU), and then passes through Sigmoid to generate channel attention weight Mc, and then Mc is multiplied by the input feature F to obtain the channel refined feature F*; F* then enters the spatial attention module, respectively performs maximum pooling and average pooling in the spatial dimension to obtain two feature maps, which are spliced through a convolution layer, and then pass through Sigmoid to generate spatial attention weight Ms, and then Ms is multiplied by F* to obtain the final refined feature output.

[0128] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced, without departing from the spirit and scope of the technical solutions of the present application, which should be covered in the scope of the claims of the present application.

Claims

1. A weld seam ROI segmentation method based on active and passive vision fusion, characterized in that, include: Simultaneously acquire two-dimensional images and three-dimensional point clouds of the surface of the workpiece to be welded; The two-dimensional image is input into a trained deep learning network, which outputs candidate weld regions and their pixel-level mask images. A cross-modal mapping relationship between the 3D point cloud and the 2D image is established through geometric calibration. The 3D point cloud is then projected onto the 2D image in a unified coordinate system to complete registration and fusion. Based on the bounding box of the candidate weld region in the pixel coordinate system, the candidate point cloud region is cropped from the three-dimensional point cloud using the cross-modal mapping relationship; Generate a grayscale matrix for ray parity determination based on the pixel-level mask image; Within the candidate point cloud region, each point of the candidate point cloud is projected to pixel coordinates. Based on the grayscale matrix, the projected points are subjected to ray parity discrimination. Points that meet the parity rules are identified as weld seam points and aggregated into a weld seam ROI point cloud. The weld seam ROI point cloud is then output as the segmentation result.

2. The weld seam ROI segmentation method based on active and passive vision fusion according to claim 1, characterized in that, The two-dimensional image is processed using a deep learning network, which is a pre-trained YOLOv8n-SEG deep learning model with an attention mechanism introduced. This deep learning network model includes two parts: object detection and instance segmentation. In the target detection stage, the detection box, category, confidence score, and coordinate information of the candidate weld region in the two-dimensional image are output. During the instance segmentation stage, a weld mask image matching the size of the original image is output.

3. The weld seam ROI segmentation method based on active and passive vision fusion according to claim 2, characterized in that, The target detection and instance segmentation specifically include: In the object detection stage, feature maps at three scales, N3 (1,256,80,80), N2 (1,512,40,40), and N1 (1,1024,20,20), are fed into the decoupled detection head. Each feature map is first processed by a 3×3 convolution, and then by two 1×1 convolutions to predict bounding boxes (4 parameters, center point coordinates x, y and dimensions w and h) and class probability boxes (n parameters). Finally, a total of 8400 candidate boxes are predicted across the three scales, with each box having (4+n) parameters. In the instance segmentation stage, convolutional upsampling is applied to N2 to generate a prototype mask, outputting (1, 32, 160, 160). For each detection box retained by the final object detection, classification-related features are extracted from its corresponding (4+n)-dimensional vector to generate 32 mask coefficients. The 32 mask coefficients of each instance are multiplied by matrix and activated with 32 160x160 prototype masks to generate a coarse-grained mask for that instance. Subsequently, the region of interest is cropped according to its bounding box coordinates, and this mask is upsampled back to the original image size of 1920x1200 to obtain the final instance segmentation result.

4. The weld seam ROI segmentation method based on active and passive vision fusion according to claim 2 or 3, characterized in that, The introduction of an attention mechanism into deep learning networks includes: The attention mechanism consists of a channel attention mechanism and a spatial attention mechanism; The mathematical formula for the channel attention mechanism is expressed as follows: M c (F)=σ{MLP[AvgPooL(F)]+MLP[MaxPooL(F)]} The mathematical formula for the spatial attention mechanism is as follows: M s (F)=σ{f 7×7 [AvgPooL(F);MaxPooL(F)]} Where F is the input feature map, MLP represents the multilayer perceptron operation, AvgPooL and MaxPool are the channel-dimensional global max pooling and global average pooling operations on F, respectively, σ represents the Sigmoid activation function, and f 7×7 It is a 7×7 convolution.

5. The weld seam ROI segmentation method based on active and passive vision fusion according to claim 1, characterized in that, Obtaining the cross-modal mapping relationship includes: Based on synchronously acquired 2D images and 3D point clouds, its cross-modal mapping is performed by the following formula: Among them, (X) i Y i Z i Let p be any point in the point cloud. i Coordinates, β is the coordinate normalization coefficient, M1 and M2 are the camera intrinsic and extrinsic parameter matrices respectively, (u i ,v i ) is p i The pixel coordinates of the point, where i∈N+ is the point number.

6. The weld seam ROI segmentation method based on active and passive vision fusion according to claim 1 or 5, characterized in that, Based on the candidate weld region, candidate point cloud regions are obtained by cropping from the 3D point cloud using the cross-modal mapping relationship, including: Based on the pixel coordinate information of the candidate region, the three-dimensional points that fall within the candidate region of the weld seam in the two-dimensional image after projection are extracted and used as the candidate point cloud region.

7. The weld seam ROI segmentation method based on active and passive vision fusion according to claim 1, characterized in that, Generate a grayscale matrix for ray parity determination based on the pixel-level mask image, including: Based on the candidate weld regions, a pixel grayscale matrix Lp is constructed, and the length n and width m of the matrix are determined according to the following formula: At this point, the grayscale matrix Lp is represented by the following formula: Where " is the floor operator, p x With p y These represent the X and Y dimensions of the pixel, respectively. The element values ​​of the Lp matrix are 255 or 0, where 255 is the background grayscale value and 0 corresponds to the grayscale value of the mask outline.

8. The weld seam ROI segmentation method based on active and passive vision fusion according to claim 1 or 7, characterized in that, Output the weld ROI point cloud as a segmentation result, including: Based on the cross-modal mapping relationship, each point of the candidate point cloud is projected to pixel coordinates, and multiple rays are emitted from each projection point outwards from the image along a predetermined direction. The number of intersections between each ray and the boundary pixel is counted on the weld mask boundary defined by the gray-scale matrix. Determine the inclusion of a point domain based on the aforementioned parity rule: When the number of intersections is odd or the proportion of rays with an odd number of intersections is greater than 50%, the point is marked as a weld point. Otherwise, mark the point as a non-weld point; Points marked as weld seams are aggregated into a weld ROI point cloud, and the weld ROI point cloud is output as the segmentation result.

9. The weld seam ROI segmentation method based on active and passive vision fusion according to claim 8, characterized in that, The candidate point cloud regions are traversed and filtered, specifically including: Select any three-dimensional point (x) in the candidate point cloud region n ,y n ,z n ), and convert them into coordinates (u) according to the cross-modal mapping relationship. n ,v n ); With the coordinates (u) n ,v n Starting from point l, emit one ray each in the vertically upward direction l and the horizontally leftward direction w, denoted as l1 and w1 respectively; Count the number of pixels with a value of 0 on each ray; If the number of pixels with a value of 0 on both rays l1 and w1 is odd, then the three-dimensional point (x n ,y n ,z n () represents the weld seam point cloud; If the number of pixels with a value of 0 on rays l1 and w1 is 50% odd and 50% even, then introduce a vertically downward ray l2 and calculate the number of pixels with a value of 0 on ray l2. If the number is odd, then the point is the weld point cloud; otherwise, it is the background point cloud. Select any 3D point in the candidate point cloud region without repetition, and repeat the above process until the candidate point cloud is traversed to complete the accurate segmentation of the weld ROI.

10. The weld seam ROI segmentation method based on active and passive vision fusion according to claim 1, characterized in that, The dataset used for pre-training is a pre-labeled dataset of weld seam images in different shapes and scenarios. It is divided into a training set, a test set, and a validation set, and trained in a ratio of 7:2:1.

Images