Weld joint extraction method based on multi-dimensional information fusion

Abstract

The invention provides a weld joint extraction method based on multi-dimensional information fusion. The method comprises the following steps: S1, collecting a Texture image and depth data of a weld joint; s2, inputting the Texture image into an image recognition model to obtain a mask image of the welding seam; s3, binarization processing is carried out on the mask image, and pixel coordinates of the welding seam are extracted; s4, aligning the pixel coordinates with the depth data to generate an initial welding seam point cloud; s5, the initial welding seam point cloud is processed through plane segmentation and a fitting algorithm, and welding seam feature points are extracted; and S6, obtaining an accurate welding seam point cloud by using a straight line fitting algorithm or a plane intersection fitting algorithm. By combining the two-dimensional image segmentation and the three-dimensional point cloud processing technology, the problem that accurate three-dimensional information extraction and background noise interference resistance of the weld joint are difficult to consider simultaneously in a complex industrial scene by a traditional method is solved, the precision and robustness of weld joint extraction are improved, the processing efficiency is ensured, and the method is suitable for popularization and application. And a high-reliability three-dimensional path planning basis is provided for an automatic welding system.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for weld seam extraction based on multi-dimensional information fusion. Background Technology

[0002] With the rapid development of intelligent manufacturing, the welding industry is transforming from automation to intelligentization, with welding robots gradually replacing traditional manual welding. However, traditional welding robots typically rely on manual teaching and programming, requiring reprogramming for different workpieces, which places high demands on operator skills and leads to decreased production efficiency. Furthermore, robots can only weld along preset paths, making it difficult to guarantee weld accuracy. To address these issues, vision-guided intelligent welding technology has emerged.

[0003] Vision-guided intelligent welding technology is widely used in aerospace, automobile manufacturing and other fields, and weld seam extraction is the key to automated welding. Traditional methods mainly include two-dimensional image processing and three-dimensional point cloud analysis. Two-dimensional image processing methods obtain position information through weld seam images, but are limited by planar information and have difficulty capturing the complex three-dimensional information of the weld seam. Three-dimensional point cloud methods can capture depth information, but have difficulty removing the complex background noise captured by structured light cameras. Summary of the Invention

[0004] This invention proposes a weld seam extraction method based on multi-dimensional information fusion, which solves the problems of existing weld seam extraction methods being either limited by planar information and unable to capture the complex three-dimensional information of the weld seam, or unable to remove the complex background noise captured by the structured light camera.

[0005] The technical solution of this invention is implemented as follows: This invention provides a weld seam extraction method based on multi-dimensional information fusion, comprising the following steps: S1, uses a structured light camera to acquire texture images and depth data of the weld; S2, Input the Texture image into the image recognition model to obtain the mask image of the weld; S3, perform binarization processing on the mask image to extract the pixel coordinates of the weld; S4, Align the pixel coordinates of the weld with the depth data to generate an initial weld point cloud; S5, The initial weld point cloud is processed using a plane segmentation algorithm and a fitting algorithm to extract weld feature points; S6. Using a straight line fitting algorithm or a plane intersection fitting algorithm, an accurate weld point cloud is obtained.

[0006] Specifically, in step S2, the image recognition model is the YOLOv11-seg model, and the method for obtaining the weld mask image through the YOLOv11-seg model includes: S21, The collected Texture image is preprocessed and then input into the backbone network of the YOLOv11-seg model to extract the hierarchical features of the Texture image. S22, by fusing features extracted from different levels of the backbone network through the neck network of the YOLOv11-seg model, takes into account information at different scales; S23, input the features fused from the neck network into the segmentation head of the YOLOv11-seg model to obtain the weld seam segmentation mask image.

[0007] Preferably, a Meta-TSSA module is introduced into the backbone network and the neck network to replace the pairwise similarity calculation of the original self-attention mechanism with second-order statistics of token features. Specifically, it includes the following steps: Based on the variational optimization framework of maximum coding rate reduction, a feature interaction paradigm is constructed to realize feature transformation. The feature transformation formula is as follows: ; in, Indicates the first k The second-order statistical vector of each attention head reflects the squared weighted average of the projected features, with dimension [missing information]. p ×1, p For projection dimensions; Indicates the first k The projection matrix of each attention head is used to project high-dimensional features. Z Project onto a lower-dimensional subspace; Represents the projected matrix Square each element in the array; Indicates the first k Grouping weight vectors for each attention head, express The inner product with a vector of all 1s; Design a dynamic feature weighting mechanism to achieve adaptive feature selection using the following formula: ; in, Indicates the first k A diagonal matrix of attention heads; This represents the operation of converting an input vector into a diagonal matrix; This means taking the reciprocal of each element in the vector; A feature fusion formula is constructed to integrate global contextual information. The fusion formula is as follows: ; in, Indicates the first The token feature vector is updated layer by layer; Indicates the first The token feature vector input to the layer; K The total number of attention heads.

[0008] Preferably, a Focal-Modulation module is introduced into the backbone network to replace the original SPPF module. The Focal-Modulation module uses hierarchical context encoding and gated aggregation mechanisms to dynamically integrate multi-scale contextual information, and generates the output representation through element-wise interaction between the modulator and the query features. Specifically, it includes the following steps: A hierarchical feature representation is constructed using depthwise separable convolutions, as shown in the following expression: ; in, For the first A hierarchical feature map represents the feature mapping after hierarchical contextualization. For the first A level-based contextualization function used for features from the previous level. Processing is performed; GeLU is the activation function used to introduce non-linearity; DW-Conv is a depthwise separable convolution operation; A gated aggregation mechanism is used to dynamically aggregate features at each level. The aggregation formula is as follows: ; in, The aggregated feature is the output feature obtained by weighted aggregation of features at each level. L The number of levels in the hierarchical processing; Indicates the first Level-based gating weights; This represents the element-wise multiplication operation; Feature representation is enhanced through element-level modulation, and the feature modulation formula is as follows: ; in, For the first i A concise representation of a query token; This is a query projection function used to extract the feature vector at a specific location in the feature map. Linear projection is the query vector; For feature map location i The eigenvector at that location; As a feature modulator, it aggregates features The result is obtained through lightweight processing and is used for query vectors. Perform feature enhancement. X This is the input feature map.

[0009] Preferably, a task-dynamic alignment segmentation head is introduced to replace the original segmentation head. Through task decomposition and feature alignment strategies, the segmentation accuracy of the weld area is improved. Specifically, the steps include: By using shared convolutional layers, feature extraction and dimensionality reduction are performed on the feature map output by the neck network to obtain shared features. The shared features are decomposed using the task decomposition module, which processes them through the following three branches: Classification branch: Generates categorical decomposition features for target classification; Regression branch: Generates bounding box regression decomposition features for target localization; Prototype branch: Generates prototype decomposition features, which are used for subsequent mask generation; Offsets and masks are generated based on the prototype decomposition features. The offsets and masks are then aligned with the prototype decomposition features using a dynamic convolution alignment module to generate the final mask prediction result. The category decomposition features are input into the category probability generation module to generate the category probability of each pixel, thus obtaining the category prediction result. Input the regression decomposition features into the bounding box generation module to generate bounding box prediction results; The mask prediction results, category prediction results, and bounding box prediction results are scaled and concatenated to obtain the final detection box and segmentation mask.

[0010] Specifically, in step S4, the method for generating the initial weld point cloud includes the following steps: S41, coordinate alignment: For each pixel ,like and Then the three-dimensional coordinates of the pixel are calculated as follows: ; in, C This is the set of pixel coordinates of the weld seam extracted in step S3; at pixel The depth value is obtained from the depth data; For pixels 3D coordinates; This is the reprojection mapping vector, obtained from the intrinsic parameters of the structured light camera; like or Let the three-dimensional coordinates of that point be... ; S42, Point Cloud Construction: All pixels Corresponding three-dimensional coordinate points Add it to the point cloud set to obtain the initial weld point cloud.

[0011] Specifically, in step S5, the method for extracting weld feature points includes the following steps: S51, Plane Segmentation and Fitting: The initial weld point cloud was segmented using a region segmentation algorithm to obtain two planar regions. The RANSAC plane fitting algorithm was used to fit the two planar regions, resulting in two plane equations: ; in, , Let the normal vectors of the two planes be respectively represented in the diagram. x , y , z Components along the axial direction; , These represent the intercepts of the two planes, respectively. Calculate the direction vector of the weld: ; in, s The direction vector of the weld; S52, Extraction of weld feature points: make For any given value of zero, solve the plane equation to obtain a point on the weld line. ; Calculate a data point in the weld point cloud q Distance between the lines of intersection of the two planes : ; in, Point q The projection point on the weld line, t Point q Projection parameters onto the weld line; Set threshold ,like Then point q For weld features, extract all welds that meet the threshold condition. Weld feature points.

[0012] Optionally, in step S6, the extracted weld feature points are fitted with straight lines using the RANSAC straight line fitting method, and a precise weld point cloud is obtained by uniform interpolation.

[0013] Preferably, in step S6, the two planar regions obtained by the planar segmentation in step S5 are subjected to planar intersection fitting using the planar intersection fitting method, and a precise weld point cloud is obtained by uniform interpolation.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) By combining two-dimensional image segmentation and three-dimensional point cloud processing technology, this invention effectively solves the problem that traditional methods are difficult to simultaneously extract accurate three-dimensional information of welds and resist background noise interference in complex industrial scenarios. It significantly improves the accuracy and robustness of weld extraction, while ensuring processing efficiency, and provides a highly reliable three-dimensional path planning basis for automated welding systems. (2) This invention introduces the Meta-TSSA module into the backbone network and neck network of the original YOLOv11-seg model. The Meta-TSSA module uses a second-order statistical feature interaction mechanism to replace the traditional self-attention calculation, which greatly reduces the computational complexity, enables the model to efficiently process high-resolution images, and enhances the ability to capture weld edges and texture features, thereby improving the continuity of the segmentation results. (3) This invention replaces the original SPPF module by introducing the Focal-Modulation module into the backbone network of the original YOLOv11-seg model. Focal-Modulation achieves adaptive fusion of context information through multi-scale feature encoding and gating aggregation mechanism, and can dynamically enhance key regions according to local features through element-level modulation operation, effectively suppressing background noise and significantly improving segmentation stability under complex working conditions. (4) This invention replaces the original segmentation head by introducing a task dynamic alignment segmentation head into the original YOLOv11-seg model. Through task decomposition and feature alignment strategies, the classification, regression and mask generation tasks are optimized respectively. The dynamic convolution alignment module effectively solves the problem of weld boundary ambiguity. The generated spatial adaptive features significantly improve the positioning accuracy of the mask boundary. (5) By establishing a precise mask coordinate and depth data alignment mechanism, the initial weld point cloud generated by this invention fully preserves the three-dimensional structural features of the weld; subsequently, by combining planar segmentation and fitting strategies, point cloud noise is effectively filtered out and high-precision weld feature points are extracted, and finally, a high-precision weld point cloud that meets the requirements of automated welding is output. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a weld seam extraction method based on multi-dimensional information fusion according to the present invention. Figure 2 This is a schematic diagram of the structure of the traditional YOLOv11-seg model; Figure 3 This is a schematic diagram of the structure of the improved YOLOv11-seg model in an embodiment of the present invention; Figure 4 This is a schematic diagram of the Meta-TSSA module in an embodiment of the present invention; Figure 5 This is a schematic diagram of the Focal-Modulation module in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of the task dynamic alignment segmentation head in an embodiment of the present invention; Figure 7 This is a schematic diagram of the overall process for weld seam extraction in an embodiment of the present invention; Figure 8 This is a rendering of the initial weld point cloud obtained in an embodiment of the present invention; Figure 9 This is a diagram showing the effect of planar segmentation of the initial weld point cloud in an embodiment of the present invention. Figure 10 This is a rendering of the weld feature point cloud extracted in an embodiment of the present invention; Figure 11 This is a rendering of the weld point cloud obtained by two linear fitting methods in an embodiment of the present invention. Figure 12 This is an error map of the weld point cloud obtained using the RANSAC linear fitting method in an embodiment of the present invention. Figure 13 This is an error map of the weld point cloud obtained using the planar intersection fitting method in an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] Reference Figure 1 This invention provides a weld seam extraction method based on multi-dimensional information fusion, comprising the following steps: S1, uses a structured light camera to acquire texture images and depth data of the weld; S2, Input the Texture image into the image recognition model to obtain the mask image of the weld; S3, perform binarization processing on the mask image to extract the pixel coordinates of the weld; S4, Align the pixel coordinates of the weld with the depth data to generate an initial weld point cloud; S5, The initial weld point cloud is processed using a plane segmentation algorithm and a fitting algorithm to extract weld feature points; S6. Using a straight line fitting algorithm or a plane intersection fitting algorithm, an accurate weld point cloud is obtained.

[0019] Specifically, such as Figure 2 As shown, in step S2, the image recognition model is the YOLOv11-seg model, and the method for obtaining the weld mask image through the YOLOv11-seg model includes: S21, The collected Texture image is preprocessed and then input into the backbone network of the YOLOv11-seg model to extract the hierarchical features of the Texture image. S22, by fusing features extracted from different levels of the backbone network through the neck network of the YOLOv11-seg model, takes into account information at different scales; S23, input the features fused from the neck network into the segmentation head of the YOLOv11-seg model to obtain the weld seam segmentation mask image.

[0020] Preferably, such as Figure 3 , 4 As shown, a Meta-TSSA module is introduced into the backbone and neck networks to replace the pairwise similarity calculation of the original self-attention mechanism with second-order statistics of token features. Specifically, this includes the following steps: Based on the variational optimization framework of maximum coding rate reduction, a feature interaction paradigm is constructed to realize feature transformation. The feature transformation formula is as follows: ; in, Indicates the first k The second-order statistical vector of each attention head reflects the squared weighted average of the projected features, with dimension [missing information]. p ×1, p For projection dimensions; Indicates the first k The projection matrix of each attention head is used to project high-dimensional features. Z Project onto a lower-dimensional subspace; Represents the projected matrix Square each element in the array; Indicates the first k Grouping weight vectors for each attention head, express The inner product with a vector of all 1s; Design a dynamic feature weighting mechanism to achieve adaptive feature selection using the following formula: ; in, Indicates the first k A diagonal matrix of attention heads; This represents the operation of converting an input vector into a diagonal matrix; This means taking the reciprocal of each element in the vector; A feature fusion formula is constructed to integrate global contextual information. The fusion formula is as follows: ; in, Indicates the first The token feature vector is updated layer by layer; Indicates the first The token feature vector input to the layer; K The total number of attention heads.

[0021] The Meta-TSSA module achieves linear complexity (O(...)) through its design. pn ) time, O( p YOLOv11-seg enables fast inference in high-resolution image processing by capturing global dependencies, thereby enhancing the segmentation accuracy of weld edges and textures. Meanwhile, its modular structure is seamlessly integrated with the backbone and neck networks of YOLOv11-seg, optimizing multi-scale feature fusion, thus improving the performance and efficiency of YOLOv11-seg in weld segmentation tasks.

[0022] In this embodiment, the structural diagram of the Meta-TSSA module is as follows: Figure 4 As shown, it specifically includes: 1) LayerNorm is used to normalize the d-dimensional features of each token, stabilize the training process, and enhance the robustness of the model to changes in the input distribution. Input: The token sequence after flattening the input feature map and transforming it through linear projection; Output: The normalized sequence of tokens; 2) Multilayer Perceptron (MLP) enhances the expressive power of the model through nonlinear transformations (such as fully connected layers and activation functions), captures complex feature patterns, and the output participates in residual connections to integrate global information; Input: A normalized sequence of tokens; Output: A sequence of tokens after non-linear transformation; 3) Token statistics are derived from the attention module Token-statics-Self-Attention, which is used to calculate the global dependency relationship between tokens and capture long-distance contextual information; Input: A normalized sequence of tokens; Output: A sequence of tokens processed by the self-attention mechanism; 4) Diagonal Attention Head: Through multiple diagonal attention iterations, it captures local dependencies at different scales, enhancing the model's adaptability to complex structures (such as weld edges). Input: A normalized sequence of tokens; Output: The token sequence after diagonal attention processing.

[0023] Preferably, such as Figure 3 , 5 As shown, a Focal-Modulation module is introduced into the backbone network to replace the original SPPF module. The Focal-Modulation module uses hierarchical context encoding and gated aggregation mechanisms to dynamically integrate multi-scale contextual information, and generates the output representation through element-wise interaction between the modulator and the query features. Specifically, it includes the following steps: A hierarchical feature representation is constructed using depthwise separable convolutions, as shown in the following expression: ; in, For the first A hierarchical feature map represents the feature mapping after hierarchical contextualization. For the first A level-based contextualization function used for features from the previous level. Processing is performed; GeLU is the activation function used to introduce non-linearity; DW-Conv is a depthwise separable convolution operation; A gated aggregation mechanism is used to dynamically aggregate features at each level. The aggregation formula is as follows: ; in, The aggregated feature is the output feature obtained by weighted aggregation of features at each level. L The number of levels in the hierarchical processing; Indicates the first Level-based gating weights; This represents the element-wise multiplication operation; Feature representation is enhanced through element-level modulation, and the feature modulation formula is as follows: ; in, For the first i A concise representation of a query token; This is a query projection function used to extract the feature vector at a specific location in the feature map. Linear projection is the query vector; For feature map location i The eigenvector at that location; As a feature modulator, it aggregates features The result is obtained through lightweight processing and is used for query vectors. Perform feature enhancement. X This is the input feature map.

[0024] In this embodiment, the Focal-Modulation module linearly transforms the input features through a fully connected layer, employs hierarchical context encoding and gated aggregation mechanisms to dynamically integrate multi-scale contextual information, and finally generates the output representation through element-wise interaction between the modulator and the query features. This improves the model's segmentation accuracy and provides a better spatial feature extraction scheme for weld seam segmentation tasks.

[0025] The structural diagram of the Focal-Modulation module is shown below. Figure 5 As shown, it specifically includes: 1) Linear layers are used to provide the initial feature representation of the network as input for subsequent modules; Input: Initial feature map, dimension ( C , H , W ),in, C The initial number of feature map channels. H , W The height and width of the feature map; Output: Feature map after linear transformation, with dimension ( d , H , W ),in, d This represents the number of channels in the feature map after linear projection. 2) The aggregation layer is used to perform preliminary spatial or channel-level aggregation on the input features, extracting contextual information for the modulator and query vector. q Provide basic features; Input: Feature map output from a linear layer; Output: Aggregated feature map; 3) Modulator, used to generate dynamic modulators, captures contextual information related to the query vector through hierarchical context (from short-range to long-range, including global) and gating mechanisms; Input: Feature map output from the aggregation layer; Output: A dynamic modulator is generated based on multi-level contextualization (combining hierarchical contextualization and gated aggregation). .

[0026] Preferably, such as Figure 3 , 6 As shown, a task-dynamic alignment segmentation head (TDAS-Head) is introduced to replace the original segmentation head. Through task decomposition and feature alignment strategies, the segmentation accuracy of the weld area is improved. Specifically, the steps include: By using shared convolutional layers, feature extraction and dimensionality reduction are performed on the feature map output by the neck network to obtain shared features. The shared features are decomposed using the task decomposition module, which processes them through the following three branches: Classification branch: Generates categorical decomposition features for target classification; Regression branch: Generates bounding box regression decomposition features for target localization; Prototype branch: Generates prototype decomposition features, which are used for subsequent mask generation; Offsets and masks are generated based on the prototype decomposition features. The offsets and masks are then aligned with the prototype decomposition features using a dynamic convolution alignment module to generate the final mask prediction result. The category decomposition features are input into the category probability generation module to generate the category probability of each pixel, thus obtaining the category prediction result. Input the regression decomposition features into the bounding box generation module to generate bounding box prediction results; The mask prediction results, category prediction results, and bounding box prediction results are scaled and concatenated to obtain the final detection box and segmentation mask.

[0027] In this embodiment, the structure diagram of the Task Dynamic Alignment Segmentation Head (TDAS-Head) is as follows: Figure 6 As shown, it specifically includes: 1) Conv_GN, a convolutional normalization layer, is part of a shared convolutional layer. It extracts initial features through convolution and normalization (GN), halves the number of channels to reduce computational cost, and provides a high-quality shared representation for subsequent task decomposition. Input: Feature map output by the neck network; Output: Initial feature map (shared features) after convolution and normalization to halve the number of channels.

[0028] 2) The TaskDecomposition module is used to decompose shared features into features for classification, regression, and segmentation masking tasks, and enhances task relevance through convolution and pooling. Input: Shared features output from the Conv_GN convolutional normalization layer; Output: categorical decomposition features, regression decomposition features, and prototype decomposition features. Each branch contains two convolutional layers and an adaptive average pooling layer.

[0029] 3) Mask & offset generator, used to generate offsets and masks based on prototype decomposition features; Input: Prototype decomposition features output by the TaskDecomposition module; Output: offset and mask.

[0030] 4) The dynamic convolutional alignment module Multiply is used to combine prototype features with dynamically generated offsets and masks to generate the final mask, improving the adaptability of segmentation tasks; Input: Prototype features, offsets, and mask; Output: Adjusted features (final mask prediction result).

[0031] 5) The bounding box generation module conv_reg is used to process regression features and generate bounding box parameters (position and size), providing a basis for target localization; Input: Regression decomposition features; Output: Regression prediction features.

[0032] 6) The category probability generation module conv-Relu-conv-Sigmoid is used to process category features and generate the category probability at each position. It improves classification accuracy through non-linear activation and normalization. Input: Class decomposition features; Output: Class probability.

[0033] 7) The category prediction module conv_cls is used to refine the category probabilities, generate the final classification results, and enhance the ability to distinguish between categories; Input: Class probabilities output by the conv-Relu-conv-Sigmoid layer; Output: Final category prediction results.

[0034] 8) The scaling module, scale, is used to scale classification, regression, and masking results; Input: Regression prediction features, final category prediction results, and mask prediction results; Output: Scaled detection and segmentation results.

[0035] Specifically, in step S3, the method for binarizing the mask image is as follows: set the RGB value of the mask area (i.e., the weld area) to 255, and set the RGB value of the non-mask area (i.e., the background area) to 0.

[0036] Specifically, in step S4, the method for generating the initial weld point cloud includes the following steps: S41, coordinate alignment: For each pixel ,like and Then the three-dimensional coordinates of the pixel are calculated as follows: ; in, C This is the set of pixel coordinates of the weld seam extracted in step S3; at pixel The depth value is obtained from the depth data; For pixels 3D coordinates; This is the reprojection mapping vector, obtained from the intrinsic parameters of the structured light camera; like or Let the three-dimensional coordinates of that point be... ; S42, Point Cloud Construction: All pixels Corresponding three-dimensional coordinate points Add it to the point cloud set to obtain the initial weld point cloud.

[0037] Specifically, in step S5, the method for extracting weld feature points includes the following steps: S51, Plane Segmentation and Fitting: The initial weld point cloud was segmented using a region segmentation algorithm to obtain two planar regions. The RANSAC plane fitting algorithm was used to fit the two planar regions, resulting in two plane equations: ; in, , Let the normal vectors of the two planes be respectively represented in the diagram. x , y , z Components along the axial direction; , These represent the intercepts of the two planes, respectively. Calculate the direction vector of the weld: ; in, sThe direction vector of the weld; S52, Extraction of weld feature points: make For any given value of zero, solve the plane equation to obtain a point on the weld line. ; Calculate a data point in the weld point cloud q Distance between the lines of intersection of the two planes : ; in, Point q The projection point on the weld line, t Point q Projection parameters onto the weld line; Set threshold ,like Then point q For weld features, extract all welds that meet the threshold condition. Weld feature points.

[0038] Optionally, in step S6, the extracted weld feature points are fitted with straight lines using the RANSAC straight line fitting method, and a precise weld point cloud is obtained by uniform interpolation.

[0039] Preferably, in step S6, the two planar regions obtained by the planar segmentation in step S5 are subjected to planar intersection fitting using the planar intersection fitting method, and a precise weld point cloud is obtained by uniform interpolation.

[0040] The overall process of weld seam extraction in this invention is as follows: Figure 7 As shown, the initial weld point cloud is obtained through steps S1-S4. Figure 8 As shown, the planar segmentation result obtained through step S5 is as follows: Figure 9 As shown, Figure 9 The yellow and blue areas represent the two planes obtained through segmentation; the set of weld feature points (point cloud) extracted in step S5 is as follows: Figure 10 As shown, Figure 10 The pink linear region between the two planes represents the extracted weld feature point cloud; the weld point cloud obtained through the two straight-line fitting methods in step S6 is as follows. Figure 11 As shown, Figure 11 In the diagram, the red lines represent the weld point cloud obtained using the RANSAC linear fitting method, and the errors of the corresponding weld points in the X, Y, and Z directions are shown below. Figure 12As shown, the color of the weld point represents the total error of the weld point in three directions, corresponding to the error color distribution column on the right. The maximum errors in the X, Y, and Z directions are 1.797mm, 1.883mm, and 1.366mm, respectively, and the average errors in the three directions are 1.105mm, 1.648mm, and 0.857mm, respectively. The overall maximum and average errors are 2.362mm and 2.221mm, respectively. The blue line represents the weld point cloud obtained using the plane intersection fitting method, and the corresponding errors of the weld points in the X, Y, and Z directions are shown below. Figure 13 As shown, the maximum errors in the X, Y, and Z directions are 0.161 mm, 1.091 mm, and 0.533 mm, respectively, and the average errors in the three directions are 0.079 mm, 0.456 mm, and 0.251 mm, respectively. The overall maximum error and average error are 1.197 mm and 0.555 mm, respectively. In summary, the positioning accuracy of the weld point cloud obtained by the plane intersection fitting method is significantly better than that obtained by the RANSAC linear fitting method, especially in terms of overall error.

[0041] In this embodiment, multiple sets of experiments are designed to compare the performance parameters of the improved YOLOv11-seg model of the present invention with the original model and other models. The performance comparison parameters between the improved YOLOv11-seg model and the original model are shown in Table 1 below: Table 1 Performance Comparison of the Improved YOLOv11-seg Model and the Original Model

[0042] As shown in Table 1, the complete model integrating the three modules Meta-TSSA, Focal-Modulation, and TDAS-Head achieved the best overall performance compared to models based on partial module integration or the original model. Compared to the basic model, its mAP75 and mAP50:95 improved by 7.4% and 2.12%, respectively, which were the highest values ​​in all experiments. Simultaneously, the number of parameters decreased by 13.05%, and the model size decreased by 13.79%, indicating a more lightweight model. Furthermore, the inference speed (FPS) reached 52.19, an improvement of 8.64%. Overall, the experimental data demonstrate the effectiveness of the three modules introduced in this invention; their combination successfully achieved a better balance between accuracy, speed, and complexity.

[0043] The performance comparison parameters of the improved YOLOv11-seg model with other existing models are shown in Table 2 below: Table 2 Performance comparison of the improved YOLOv11-seg model with other models

[0044] As shown in Table 2, the improved YOLOv11-seg model of this invention achieves the best values ​​among all compared algorithms in four key metrics: mAP75, mAP50:95, and the number of parameters and model size (measuring model complexity). Compared with the YOLOv11-seg model that forms its basis, the improved YOLOv11-seg significantly improves mAP75 and mAP50:95 accuracies by 7.4% and 2.12%, respectively, while reducing the number of parameters and model size by more than 13%, and increasing the inference speed (FPS) from 48.04 to 52.19, demonstrating better inference speed. Compared with more complex models like Mamba-YOLO-seg, the improved YOLOv11-seg achieves higher segmentation accuracy and a 45% improvement in inference speed with less than half the number of parameters. Compared to algorithms such as YOLOv8-seg and YOLOv9t-seg, the improved YOLOv11-seg also achieves better segmentation accuracy while being more lightweight and having fewer parameters. Although its inference speed is slower than YOLOv12-seg, its mAP75 and mAP50:95 are increased by 8.45% and 3.47%, respectively. In summary, the improved YOLOv11-seg model proposed in this invention exhibits better overall performance than other algorithms in weld seam region segmentation tasks, while achieving an FPS of 52.19, meeting the real-time segmentation requirements in industry.

[0045] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A weld seam extraction method based on multi-dimensional information fusion, characterized in that, Includes the following steps: S1, uses a structured light camera to acquire texture images and depth data of the weld; S2, Input the Texture image into the image recognition model to obtain the mask image of the weld; S3, perform binarization processing on the mask image to extract the pixel coordinates of the weld; S4, Align the pixel coordinates of the weld with the depth data to generate an initial weld point cloud; S5, The initial weld point cloud is processed using a plane segmentation algorithm and a fitting algorithm to extract weld feature points; S6. Using a straight line fitting algorithm or a plane intersection fitting algorithm, an accurate weld point cloud is obtained.

2. The weld seam extraction method based on multi-dimensional information fusion as described in claim 1, characterized in that, In step S2, the image recognition model is the YOLOv11-seg model, and the method for obtaining the weld mask image using the YOLOv11-seg model includes: S21, The collected Texture image is preprocessed and then input into the backbone network of the YOLOv11-seg model to extract the hierarchical features of the Texture image. S22, by fusing features extracted from different levels of the backbone network through the neck network of the YOLOv11-seg model, takes into account information at different scales; S23, input the features fused from the neck network into the segmentation head of the YOLOv11-seg model to obtain the weld seam segmentation mask image.

3. The weld seam extraction method based on multi-dimensional information fusion as described in claim 2, characterized in that, Meta-TSSA modules are introduced into the backbone and neck networks, replacing the pairwise similarity calculation of the original self-attention mechanism with second-order statistics of token features. The specific steps include: Based on the variational optimization framework of maximum coding rate reduction, a feature interaction paradigm is constructed to realize feature transformation. The feature transformation formula is as follows: ； in, Indicates the first k The second-order statistical vector of each attention head reflects the squared weighted average of the projected features, with dimension [missing information]. p ×1, p For projection dimensions; Indicates the first k The projection matrix of each attention head is used to project high-dimensional features. Z Project onto a lower-dimensional subspace; Represents the projected matrix Square each element in the array; Indicates the first k Grouping weight vectors for each attention head, express The inner product with a vector of all 1s; Design a dynamic feature weighting mechanism to achieve adaptive feature selection using the following formula: ； in, Indicates the first k A diagonal matrix of attention heads; This represents the operation of converting an input vector into a diagonal matrix; This means taking the reciprocal of each element in the vector; A feature fusion formula is constructed to integrate global contextual information. The fusion formula is as follows: ； in, Indicates the first The token feature vector is updated layer by layer; Indicates the first The token feature vector input to the layer; K The total number of attention heads.

4. The weld seam extraction method based on multi-dimensional information fusion as described in claim 2, characterized in that, A Focal-Modulation module is introduced into the backbone network to replace the original SPPF module. The Focal-Modulation module uses hierarchical context encoding and gated aggregation mechanisms to dynamically integrate multi-scale context information, and generates the output representation through element-wise interaction between the modulator and query features. Specifically, it includes the following steps: A hierarchical feature representation is constructed using depthwise separable convolutions, as shown in the following expression: ； in, For the first A hierarchical feature map represents the feature mapping after hierarchical contextualization. For the first A level-based contextualization function used for features from the previous level. Processing is performed; GeLU is the activation function used to introduce non-linearity; DW-Conv is a depthwise separable convolution operation; A gated aggregation mechanism is used to dynamically aggregate features at each level. The aggregation formula is as follows: ； in, The aggregated feature is the output feature obtained by weighted aggregation of features at each level. L The number of levels in the hierarchical processing; Indicates the first Level-based gating weights; This represents the element-wise multiplication operation; Feature representation is enhanced through element-level modulation, and the feature modulation formula is as follows: ； in, For the first i A concise representation of a query token; This is a query projection function used to extract the feature vector at a specific location in the feature map. Linear projection is the query vector; For feature map location i The eigenvector at that location; As a feature modulator, it aggregates features The result is obtained through lightweight processing and is used for query vectors. Perform feature enhancement. X This is the input feature map.

5. The weld seam extraction method based on multi-dimensional information fusion as described in claim 2, characterized in that, A task-dynamic alignment segmentation head is introduced to replace the original segmentation head. Through task decomposition and feature alignment strategies, the segmentation accuracy of the weld area is improved. The specific steps include: By using shared convolutional layers, feature extraction and dimensionality reduction are performed on the feature map output by the neck network to obtain shared features. The shared features are decomposed using the task decomposition module, which processes them through the following three branches: Classification branch: Generates categorical decomposition features for target classification; Regression branch: Generates bounding box regression decomposition features for target localization; Prototype branch: Generates prototype decomposition features, which are used for subsequent mask generation; Offsets and masks are generated based on the prototype decomposition features. The offsets and masks are then aligned with the prototype decomposition features using a dynamic convolution alignment module to generate the final mask prediction result. The category decomposition features are input into the category probability generation module to generate the category probability of each pixel, thus obtaining the category prediction result. Input the regression decomposition features into the bounding box generation module to generate bounding box prediction results; The mask prediction results, category prediction results, and bounding box prediction results are scaled and concatenated to obtain the final detection box and segmentation mask.

6. The weld seam extraction method based on multi-dimensional information fusion as described in claim 1, characterized in that, In step S4, the method for generating the initial weld point cloud includes the following steps: S41, coordinate alignment: For each pixel ,like and Then the three-dimensional coordinates of the pixel are calculated as follows: ； in, C This is the set of pixel coordinates of the weld seam extracted in step S3; at pixel The depth value is obtained from the depth data; For pixels 3D coordinates; This is the reprojection mapping vector, obtained from the intrinsic parameters of the structured light camera; like or Let the three-dimensional coordinates of that point be... ; S42, Point Cloud Construction: All pixels corresponding three-dimensional coordinate points Add it to the point cloud set to obtain the initial weld point cloud.

7. The weld seam extraction method based on multi-dimensional information fusion as described in claim 1, characterized in that, In step S5, the method for extracting weld feature points includes the following steps: S51, Plane Segmentation and Fitting: The initial weld point cloud was segmented using a region segmentation algorithm to obtain two planar regions. The RANSAC plane fitting algorithm was used to fit the two planar regions, resulting in two plane equations: ； in, , Let the normal vectors of the two planes be respectively represented in the diagram. x , y , z Components along the axial direction; , These represent the intercepts of the two planes, respectively. Calculate the direction vector of the weld: ； in, s The direction vector of the weld; S52, Extraction of weld feature points: make For any given value of zero, solve the plane equation to obtain a point on the weld line. ; Calculate a data point in the weld point cloud q Distance between the lines of intersection of the two planes : ； in, Point q The projection point on the weld line, t Point q Projection parameters onto the weld line; Set threshold ,like Then point q For weld features, extract all welds that meet the threshold condition. Weld feature points.

8. The weld seam extraction method based on multi-dimensional information fusion as described in claim 1, characterized in that, In step S6, the extracted weld feature points are fitted with straight lines using the RANSAC straight line fitting method, and a precise weld point cloud is obtained through uniform interpolation.

9. The weld seam extraction method based on multi-dimensional information fusion as described in claim 1, characterized in that, In step S6, the two planar regions obtained by the planar segmentation in step S5 are subjected to planar intersection fitting using the planar intersection fitting method, and the accurate weld point cloud is obtained by uniform interpolation.

Images