A weld seam extraction method based on multi-dimensional information fusion

Abstract

The application provides a welding seam extraction method based on multi-dimensional information fusion, comprising the following steps: S1, collecting a Texture image and depth data of a welding seam; S2, inputting the Texture image into an image recognition model to obtain a mask image of the welding seam; S3, performing binaryzation processing on the mask image to extract pixel coordinates of the welding seam; S4, aligning the pixel coordinates with the depth data to generate an initial welding seam point cloud; S5, processing the initial welding seam point cloud by using a plane segmentation and fitting algorithm to extract characteristic points of the welding seam; and S6, obtaining an accurate welding seam point cloud by using a straight line fitting algorithm or a plane intersection fitting algorithm. By combining two-dimensional image segmentation and three-dimensional point cloud processing technology, the application solves the problem that a traditional method is difficult to simultaneously consider accurate three-dimensional information extraction and anti-background noise interference of the welding seam in a complex industrial scene, improves the precision and robustness of welding seam extraction, and guarantees the processing efficiency, thereby providing a high-reliability three-dimensional path planning basis for an automatic welding system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a welding seam extraction method based on multi-dimensional information fusion. BACKGROUND

[0002] With the rapid development of intelligent manufacturing, the welding industry is transforming from automation to intelligence, and welding robots are gradually replacing traditional manual welding. However, traditional welding robots usually rely on manual teaching programming, and need to be reprogrammed for different workpieces, which requires high technical requirements for operators and leads to a decline in production efficiency. In addition, the robot can only weld according to the preset path, and it is difficult to guarantee the welding seam precision. To solve these problems, intelligent welding technology based on visual guidance has emerged.

[0003] Intelligent welding technology based on visual guidance is widely used in the fields of aerospace, automobile manufacturing, etc., and welding seam extraction is the key to automation welding. Traditional methods mainly include two-dimensional image processing and three-dimensional point cloud analysis. The two-dimensional image processing method obtains position information through the welding seam image, but is limited to planar information and is difficult to capture complex three-dimensional information of the welding seam. The three-dimensional point cloud method can capture depth information, but it is difficult to remove complex background noise captured by the structured light camera. SUMMARY

[0004] The present application provides a welding seam extraction method based on multi-dimensional information fusion, which solves the problems that the existing welding seam extraction method is limited to planar information and is difficult to capture complex three-dimensional information of the welding seam, or is difficult to remove complex background noise captured by the structured light camera.

[0005] The technical solution of the present application is as follows:

[0006] The present application provides a welding seam extraction method based on multi-dimensional information fusion, comprising the following steps:

[0007] S1, acquiring Texture image and depth data of the welding seam through a structured light camera;

[0008] S2, inputting the Texture image into an image recognition model to obtain a mask image of the welding seam;

[0009] S3, performing binaryzation processing on the mask image to extract pixel coordinates of the welding seam;

[0010] S4, aligning the pixel coordinates of the welding seam with the depth data to generate an initial welding seam point cloud;

[0011] S5, processing the initial welding seam point cloud by using a plane segmentation algorithm and a fitting algorithm to extract feature points of the welding seam;

[0012] S6, using a straight line fitting algorithm or a plane intersection fitting algorithm, to obtain an accurate weld point cloud.

[0013] Specifically, in step S2, the image recognition model is a YOLOv11-seg model, and the method for obtaining the weld mask image by the YOLOv11-seg model includes:

[0014] S21, inputting the preprocessed Texture image into the backbone network of the YOLOv11-seg model to extract hierarchical features of the Texture image;

[0015] S22, fusing the features of different levels extracted by the backbone network through the neck network of the YOLOv11-seg model, and taking into account information of different scales;

[0016] S23, inputting the fused features of the neck network into the segmentation head of the YOLOv11-seg model to obtain a weld segmentation mask image.

[0017] Preferably, a Meta-TSSA module is introduced in the backbone network and the neck network to replace the pair similarity calculation of the original self-attention mechanism with the second-order statistics of Token features, and the method includes the following steps:

[0018] Based on the maximum coding rate reduction-based variational optimization framework, a feature interaction paradigm is constructed to realize feature transformation, and the feature change formula is as follows:

[0019] ;

[0020] Wherein, represents the second-order statistical vector of the i-th attention head, reflecting the weighted average of the square sum of the projected features, and the dimension is k ×1, p is the projection dimension; p represents the projection matrix of the i-th attention head, which is used to project the high-dimensional features to a low-dimensional subspace; k represents the square operation on each element in the projected matrix Z ; represents the grouping weight vector of the i-th attention head, represents the inner product of and the all-1 vector; k

[0021] A dynamic feature weighting mechanism is designed to realize adaptive feature selection by the following formula:

[0022] ;

[0023] Wherein,​​​ Indicates the first k A diagonal matrix of attention heads; This represents the operation of converting an input vector into a diagonal matrix; This means taking the reciprocal of each element in the vector;

[0024] A feature fusion formula is constructed to integrate global contextual information. The fusion formula is as follows:

[0025] ;

[0026] in, Indicates the first The token feature vector is updated layer by layer; Indicates the first The token feature vector input to the layer; K The total number of attention heads.

[0027] Preferably, a Focal-Modulation module is introduced into the backbone network to replace the original SPPF module. The Focal-Modulation module uses hierarchical context encoding and gated aggregation mechanisms to dynamically integrate multi-scale contextual information, and generates the output representation through element-wise interaction between the modulator and the query features. Specifically, it includes the following steps:

[0028] A hierarchical feature representation is constructed using depthwise separable convolutions, as shown in the following expression:

[0029] ;

[0030] in, For the first A hierarchical feature map represents the feature mapping after hierarchical contextualization. For the first A level-based contextualization function used for features from the previous level. Processing is performed; GeLU is the activation function used to introduce non-linearity; DW-Conv is a depthwise separable convolution operation;

[0031] A gated aggregation mechanism is used to dynamically aggregate features at each level. The aggregation formula is as follows:

[0032] ;

[0033] in, The aggregated feature is the output feature obtained by weighted aggregation of features at each level. L The number of levels in the hierarchical processing; Indicates the first Level-based gating weights; represents an element-wise multiplication operation;

[0034] The feature representation is enhanced by element-level modulation, and the feature modulation formula is as follows:

[0035] ;

[0036] wherein, is a refined representation of the i-th query token; i is a query projection function used to project the feature vector of a certain position of the feature map to a query vector; is a feature vector at the position of the feature map; is a feature vector at the position of the feature map; i is a feature vector at the position of the feature map; is a feature modulator obtained by lightweight processing on the aggregated feature , used for feature enhancement on the query vector , X is an input feature map.

[0037] Preferably, a task dynamic alignment segmentation head is introduced to replace the original segmentation head, and the welding seam region segmentation accuracy is improved through a task decomposition and feature alignment strategy, specifically including the following steps:

[0038] The feature map output by the neck network is extracted and reduced in dimension through a shared convolutional layer to obtain shared features;

[0039] The shared features are decomposed by a task decomposition module, that is, the shared features are processed by the following three branches:

[0040] The classification branch generates class decomposition features for target classification;

[0041] The regression branch generates bounding box regression decomposition features for target positioning;

[0042] The prototype branch generates prototype decomposition features for subsequent generation of masks;

[0043] The offset and the mask are generated according to the prototype decomposition features, and the offset, the mask and the prototype decomposition features are aligned by a dynamic convolution alignment module to generate a final mask prediction result;

[0044] The class decomposition features are input into a class probability generation module to generate class probabilities of each pixel point to obtain a class prediction result;

[0045] The regression decomposition features are input into a bounding box generation module to generate a bounding box prediction result;

[0046] The mask prediction result, the class prediction result and the bounding box prediction result are scaled and spliced to obtain a final detection box and segmentation mask.

[0047] Specifically, in step S4, the method for generating the initial weld point cloud includes the following steps:

[0048] S41, coordinate alignment:

[0049] For each pixel ,like and Then the three-dimensional coordinates of the pixel are calculated as follows:

[0050] ;

[0051] in, C This is the set of pixel coordinates of the weld seam extracted in step S3; at pixel The depth value is obtained from the depth data; For pixels 3D coordinates; This is the reprojection mapping vector, obtained from the intrinsic parameters of the structured light camera;

[0052] like or Let the three-dimensional coordinates of that point be... ;

[0053] S42, Point Cloud Construction:

[0054] All pixels corresponding three-dimensional coordinate points Add it to the point cloud set to obtain the initial weld point cloud.

[0055] Specifically, in step S5, the method for extracting weld feature points includes the following steps:

[0056] S51, Plane Segmentation and Fitting:

[0057] The initial weld point cloud was segmented using a region segmentation algorithm to obtain two planar regions.

[0058] The RANSAC plane fitting algorithm was used to fit the two planar regions, resulting in two plane equations:

[0059] ;

[0060] in, , Let the normal vectors of the two planes be respectively represented in the diagram. x , y , z Components along the axial direction; , These represent the intercepts of the two planes, respectively.

[0061] Calculate the direction vector of the weld:

[0062] ;

[0063] wherein, s is the direction vector of the weld;

[0064] S52, weld feature point extraction:

[0065] Let any value is zero, solve the plane equation, get a point on the weld line ;

[0066] Calculate the distance between the two planes intersecting the distance q between the two planes intersecting the line:

[0067] ;

[0068] wherein, denotes the projection point of the point q on the weld line, t denotes the projection parameter of the point q to the weld line;

[0069] Set threshold , if , the point q is a weld feature, extract all weld feature points that meet the threshold condition .

[0070] Optionally, in step S6, the RANSAC line fitting method is used to fit the extracted weld feature points, and the accurate weld point cloud is obtained by uniform interpolation.

[0071] Preferably, in step S6, the plane intersection fitting method is used to fit the two plane regions obtained by plane segmentation in step S5, and the accurate weld point cloud is obtained by uniform interpolation.

[0072] Compared with the prior art, the beneficial effects of the present application are:

[0073] (1) The present application effectively solves the problem that traditional methods are difficult to simultaneously consider the accurate three-dimensional information extraction of the weld and the anti-background noise interference in complex industrial scenes by combining two-dimensional image segmentation and three-dimensional point cloud processing technology, significantly improves the precision and robustness of the weld extraction, and at the same time guarantees the processing efficiency, provides a high-reliability three-dimensional path planning basis for the automatic welding system;

[0074] ​(2) The Meta-TSSA module is introduced into the backbone network and the neck network of the original YOLOv11-seg model, the second-order statistical feature interaction mechanism is used instead of the traditional self-attention calculation, the calculation complexity is greatly reduced, the model can efficiently process high-resolution images, and the capture ability of the weld edge and the texture feature is enhanced, and the continuity of the segmentation result is improved;

[0075] (3) The Focal-Modulation module is introduced into the backbone network of the original YOLOv11-seg model to replace the original SPPF module, the Focal-Modulation realizes adaptive fusion of context information through multi-scale feature coding and gating aggregation mechanism, and dynamically enhances the key area according to the local feature through the element-level modulation operation, effectively suppresses the background noise, and significantly improves the segmentation stability under complex working conditions;

[0076] (4) The task dynamic alignment segmentation head is introduced into the original YOLOv11-seg model to replace the original segmentation head, the classification, regression and mask generation tasks are optimized through the task decomposition and feature alignment strategy, the dynamic convolution alignment module effectively solves the weld boundary blur problem, and the generated spatial adaptive feature significantly improves the positioning accuracy of the mask boundary;

[0077] (5) The accurate mask coordinate and depth data alignment mechanism is established, the generated initial weld point cloud completely retains the three-dimensional structure feature of the weld; subsequently, combined with the plane segmentation and fitting strategy, the point cloud noise is effectively filtered out and the high-precision weld feature point is extracted, and finally the high-precision weld point cloud meeting the requirements of automatic welding is output. BRIEF DESCRIPTION OF DRAWINGS

[0078] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0079] Figure 1 The flow chart of the welding seam extraction method based on multi-dimensional information fusion of the present application;

[0080] Figure 2 The structure diagram of the traditional YOLOv11-seg model;

[0081] Figure 3 The structure diagram of the improved YOLOv11-seg model in the embodiment of the present application;

[0082] Figure 4 Structure diagram of the Meta-TSSA module in the embodiment of the present application;

[0083] Figure 5 Structure diagram of the Focal-Modulation module in the embodiment of the present application;

[0084] Figure 6 Structure diagram of the task dynamic alignment segmentation head in the embodiment of the present application;

[0085] Figure 7 Overall flowchart of the weld extraction in the embodiment of the present application;

[0086] Figure 8 Effect diagram of the initial weld point cloud obtained in the embodiment of the present application;

[0087] Figure 9 Effect diagram of the initial weld point cloud after plane segmentation in the embodiment of the present application;

[0088] Figure 10 Effect diagram of the extracted weld feature point cloud in the embodiment of the present application;

[0089] Figure 11 Effect diagram of the weld point cloud obtained by two linear fitting methods in the embodiment of the present application;

[0090] Figure 12 Error diagram of the weld point cloud obtained by the RANSAC linear fitting method in the embodiment of the present application;

[0091] Figure 13 Error diagram of the weld point cloud obtained by the plane intersection fitting method in the embodiment of the present application. DETAILED DESCRIPTION

[0092] The technical solutions of the present application will be described clearly and completely below in conjunction with the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0093] With reference to Figure 1 , the present application provides a weld extraction method based on multi-dimensional information fusion, comprising the following steps:

[0094] S1, acquiring the Texture image and depth data of the weld through a structured light camera;

[0095] S2, inputting the Texture image into an image recognition model to obtain a mask image of the weld;

[0096] S3, binarizing the mask image to extract pixel coordinates of the weld;

[0097] S4, aligning the pixel coordinates of the weld with depth data to generate an initial weld point cloud;

[0098] S5, processing the initial weld point cloud using a plane segmentation algorithm and a fitting algorithm to extract weld feature points;

[0099] S6, using a straight line fitting algorithm or a plane intersection fitting algorithm to obtain an accurate weld point cloud.

[0100] Specifically, as shown in Figure 2 , in step S2, the image recognition model is a YOLOv11-seg model, and the method for obtaining the weld mask image by the YOLOv11-seg model includes:

[0101] S21, inputting the preprocessed Texture image into the backbone network of the YOLOv11-seg model to extract hierarchical features of the Texture image;

[0102] S22, fusing features of different levels extracted by the backbone network through the neck network of the YOLOv11-seg model, and taking into account information of different scales;

[0103] S23, inputting the fused features of the neck network into the segmentation head of the YOLOv11-seg model to obtain a weld segmentation mask image.

[0104] Preferably, as shown in Figure 3 , 4 , a Meta-TSSA module is introduced in the backbone network and the neck network to replace the pair similarity calculation of the original self-attention mechanism with the second-order statistics of Token features, which includes the following steps:

[0105] Based on the maximum coding rate reduction, a variational optimization framework is constructed to realize feature interaction and transformation, and the feature transformation formula is as follows:

[0106] ;

[0107] Among them, represents the second-order statistical vector of the i-th attention head, reflecting the weighted average of the square sum of the projected features, and the dimension is k ×1, p is the projection dimension; p represents the projection matrix of the i-th attention head, which is used to project high-dimensional features to a low-dimensional subspace; k Z ​​denotes the square operation on each element of the projected matrix denotes the square operation on each element of the projected matrix denotes the group weight vector of the k th attention head, denotes the inner product with the all-one vector denotes the inner product with the all-one vector

[0108] A dynamic feature weighting mechanism is designed to realize adaptive feature selection through the following formula:

[0109] ;

[0110] denotes the diagonal matrix of the th attention head k denotes the diagonal matrix of the th attention head denotes the inverse of each element in the vector

[0111] A feature fusion formula is constructed to integrate the global context information, and the fusion formula is as follows:

[0112] ;

[0113] denotes the token feature vector updated by the th layer denotes the token feature vector updated by the th layer denotes the token feature vector updated by the K th layer

[0114] The Meta-TSSA module realizes fast inference in high-resolution image processing through the design of linear complexity (O( pn ) time and O( p ) space); by capturing global dependencies, the segmentation accuracy of weld edges and textures is enhanced; at the same time, its modular structure seamlessly integrates with the backbone and neck network of YOLOv11-seg, optimizes multi-scale feature fusion, and thus improves the performance and efficiency of YOLOv11-seg in the weld segmentation task.

[0115] In this embodiment, the structural diagram of the Meta-TSSA module is as shown in Figure 4 , and specifically includes:

[0116] 1) Normalization layer LayerNorm, used for layer normalization of d-dimensional features of each token, stabilizing the training process and enhancing the robustness of the model to input distribution changes;

[0117] Input: token sequence after flattening and linear projection conversion of the input feature map

[0118] Output: Normalized token sequence

[0119] 2) Multi-Layer Perceptron (MLP), which enhances the model's representation ability through nonlinear transformation (such as fully connected layers and activation functions), capturing complex feature patterns, and outputs participating in residual connection to integrate global information;

[0120] Input: Normalized token sequence

[0121] Output: Token sequence after nonlinear transformation

[0122] 3) Token-statics-Self-Attention module, used to calculate the global dependency between tokens and capture long-distance context information;

[0123] Input: Normalized token sequence

[0124] Output: Token sequence processed by self-attention mechanism

[0125] 4) Diagonal Attention Head, which captures local dependency relationships of different scales through multiple diagonal attention iterations, enhancing the model's adaptability to complex structures such as weld edges;

[0126] Input: Normalized token sequence

[0127] Output: Token sequence processed by diagonal attention

[0128] Preferably, as shown in Figure 3 , 5 , a Focal-Modulation module is introduced in the backbone network to replace the original SPPF module. The Focal-Modulation module realizes dynamic integration of multi-scale context information through hierarchical context encoding and gated aggregation mechanism, and generates output representation through element-by-element interaction with query features through a modulator. The specific steps include:

[0129] A hierarchical feature representation is constructed using depthwise separable convolution, and the expression is as follows:

[0130] ;

[0131] wherein, is the feature map of the th level, representing the feature mapping after hierarchical contextualization processing; is the contextualization function of the th level, used to process the feature GeLU is an activation function for introducing nonlinearity; DW-Conv is a deep separable convolution operation;

[0132] The features of each level are dynamically aggregated by using a gating aggregation mechanism, and the aggregation formula is as follows:

[0133] ;

[0134] wherein, is the aggregated feature of the output, indicating the output feature obtained by weighted aggregation of the features of each level; L is the number of hierarchical processing layers; is the gating weight of the th level; represents an element-wise multiplication operation;

[0135] The feature representation is enhanced by element-level modulation, and the feature modulation formula is as follows:

[0136] ;

[0137] wherein, is the refined representation of the i th query token; is a query projection function, which is used to project the feature vector of a certain position of the feature map to a query vector; is the feature vector at the position i of the feature map; is a feature modulator, which is obtained by lightweight processing of the aggregated feature , and is used to enhance the feature of the query vector , X is the input feature map.

[0138] In this embodiment, the Focal-Modulation module linearly transforms the input features through a fully connected layer, realizes dynamic integration of multi-scale context information by using hierarchical context encoding and gating aggregation mechanism, and finally generates the output representation through element-wise interaction with the query feature through the modulator. The segmentation accuracy of the model is improved, and a more optimal spatial feature extraction scheme is provided for the weld seam segmentation task.

[0139] The structure diagram of the Focal-Modulation module is shown in Figure 5 , and specifically includes:

[0140] 1) Linear layer Linear, used to provide initial feature representation of the network as input for subsequent modules;

[0141] Input: initial feature map, dimension is (Cin, H, W) C ,H , W ), wherein, C is the initial number of feature map channels, H , W is the height and width of the feature map;

[0142] Output: the feature map after linear transformation processing, with dimensions of (Cout, H, W) d , H , W ), wherein, d is the number of channels of the feature map after linear projection;

[0143] 2) Aggregate layer, used for preliminary spatial or channel-level aggregation of input features, extracting context information, providing basic features for modulator and query vector q ;

[0144] Input: feature map output from the linear layer Linear;

[0145] Output: aggregated feature map;

[0146] 3) Modulator, used for generating dynamic modulator, capturing context information related to query vector through hierarchical context (short-range to long-range, including global) and gating mechanism;

[0147] Input: feature map output from the aggregation layer;

[0148] Output: dynamic modulator generated based on multi-level context aggregation (combined with hierarchical context modeling HierarchicalContextualization and gated aggregation Gated Aggregation) .

[0149] Preferably, as shown in Figure 3 , 6 , a task dynamic alignment segmentation head TDAS-Head is introduced to replace the original segmentation head, and the welding area segmentation accuracy is improved through task decomposition and feature alignment strategy, specifically including the following steps:

[0150] The feature map output by the neck network is extracted and reduced in dimension through a shared convolutional layer to obtain shared features;

[0151] The shared features are decomposed by a task decomposition module, i.e., the shared features are processed by the following three branches:

[0152] Classification branch: generate class decomposition features for target classification;

[0153] Regression branch: generate bounding box regression decomposition features for target positioning;

[0154] Prototype branch: generate prototype decomposition features for subsequent mask generation;

[0155] Generate offset and mask according to prototype decomposition features, align offset, mask and prototype decomposition features through dynamic convolution alignment module to generate final mask prediction result;

[0156] Input the class decomposition features into the class probability generation module to generate the class probability of each pixel point and obtain the class prediction result;

[0157] Input the regression decomposition features into the bounding box generation module to generate the bounding box prediction result;

[0158] Scale and splice the mask prediction result, the class prediction result and the bounding box prediction result to obtain the final detection frame and segmentation mask.

[0159] In this embodiment, the structure diagram of the task dynamic alignment segmentation head TDAS-Head is as shown in Figure 6 , and specifically includes:

[0160] 1) Convolution normalization layer Conv_GN, as part of the shared convolution layer, extracts initial features through convolution and normalization GroupNormalization (GN), halves the channel number to reduce the calculation cost, and provides high-quality shared representation for subsequent task decomposition;

[0161] Input: feature map output by neck network;

[0162] Output: initial feature map (shared feature) after halving the channel number through convolution and normalization.

[0163] 2) Task decomposition module TaskDecomposition, used to decompose the shared feature into features for classification, regression and segmentation mask tasks, and enhance task relevance through convolution and pooling;

[0164] Input: shared feature output by convolution normalization layer Conv_GN;

[0165] Output: class decomposition features, regression decomposition features and prototype decomposition features, each branch containing two convolution layers and an adaptive average pooling layer.

[0166] 3) Mask and offset generator Generator Mask&offset, used to generate offset and mask according to prototype decomposition features;

[0167] Input: prototype decomposition features output by task decomposition module TaskDecomposition;

[0168] Output: offset and mask.

[0169] 4) Dynamic convolution alignment module Multiply, used to combine the prototype feature with the dynamically generated offset and mask to generate the final mask, improve the adaptability of the segmentation task;

[0170] Input: prototype feature, offset and mask;

[0171] Output: adjusted feature (final mask prediction result).

[0172] 5) Boundary box generation module conv_reg, used to process regression features to generate boundary box parameters (position and size) to provide a basis for target positioning;

[0173] Input: regression decomposition feature;

[0174] Output: regression prediction feature.

[0175] 6) Category probability generation module conv-Relu-conv-Sigmoid, used to process category features to generate category probability at each position, and to improve classification accuracy through nonlinear activation and normalization;

[0176] Input: category decomposition feature;

[0177] Output: category probability.

[0178] 7) Category prediction module conv_cls, used to refine the category probability to generate the final classification result and enhance the category differentiation ability;

[0179] Input: category probability output by the conv-Relu-conv-Sigmoid layer;

[0180] Output: final category prediction result.

[0181] 8) Scaling module scale, used to scale the classification, regression and mask results;

[0182] Input: regression prediction feature, final category prediction result and mask prediction result;

[0183] Output: scaled detection and segmentation result.

[0184] Specifically, in step S3, the method of binarizing the mask image is: setting the RGB value of the mask region (i.e. the weld region) to 255, and setting the RGB value of the non-mask region (i.e. the background region) to 0.

[0185] Specifically, in step S4, the method of generating the initial weld point cloud includes the following steps:

[0186] S41, coordinate alignment:

[0187] For each pixel point , if and , the three-dimensional coordinates of the pixel point are calculated as:

[0188] ;

[0189] wherein C is the pixel coordinate set of the weld extracted in step S3; is the depth value at the pixel, obtained from the depth data; is the three-dimensional coordinates of the pixel point ; is the re-projection mapping vector, obtained from the intrinsic parameters of the structured light camera; If

[0190] or , the three-dimensional coordinates of the point are set as ;

[0191] S42, point cloud construction:

[0192] Add the three-dimensional coordinate points corresponding to all pixel points to the point cloud set to obtain the initial weld point cloud.

[0193] Specifically, in step S5, the method for extracting the weld feature points includes the following steps:

[0194] S51, plane segmentation and fitting:

[0195] Segment the initial weld point cloud using a region segmentation algorithm to obtain two plane regions;

[0196] Use the RANSAC plane fitting algorithm to fit the two plane regions to obtain two plane equations:

[0197] ;

[0198] wherein , represent the components of the normal vectors of the two planes in the x , y , z axis directions, respectively; , represent the intercepts of the two planes, respectively;

[0199] Calculate the direction vector of the weld:

[0200] ​ ;

[0201] in, s The direction vector of the weld;

[0202] S52, Extraction of weld feature points:

[0203] make For any given value of zero, solve the plane equation to obtain a point on the weld line. ;

[0204] Calculate a data point in the weld point cloud q Distance between the lines of intersection of the two planes :

[0205] ;

[0206] in, Point q The projection point on the weld line, t Point q Projection parameters onto the weld line;

[0207] Set threshold ,like Then point q For weld features, extract all welds that meet the threshold condition. Weld feature points.

[0208] Optionally, in step S6, the extracted weld feature points are fitted with straight lines using the RANSAC straight line fitting method, and a precise weld point cloud is obtained by uniform interpolation.

[0209] Preferably, in step S6, the two planar regions obtained by the planar segmentation in step S5 are subjected to planar intersection fitting using the planar intersection fitting method, and a precise weld point cloud is obtained by uniform interpolation.

[0210] The overall process for weld seam extraction in this invention is as follows: Figure 7 As shown, the initial weld point cloud is obtained through steps S1-S4. Figure 8 As shown, the planar segmentation result obtained through step S5 is as follows: Figure 9 As shown, Figure 9 The yellow and blue areas represent the two planes obtained through segmentation; the set of weld feature points (point cloud) extracted in step S5 is as follows: Figure 10 As shown, Figure 10 The pink linear region between the two planes represents the extracted weld feature point cloud; the weld point cloud obtained through the two straight-line fitting methods in step S6 is as follows. Figure 11 As shown, Figure 11In this embodiment, the red line is the weld point cloud obtained by using the RANSAC linear fitting method, and the errors of the corresponding weld points in the X, Y and Z directions are as shown in FIG. 6B. Figure 12 As shown in FIG. 6B, the color of the weld point represents the total error of the weld point in three directions, which corresponds to the error color distribution column on the right side; the maximum errors in the X, Y and Z directions are 1.797 mm, 1.883 mm and 1.366 mm respectively, and the average errors in the three directions are 1.105 mm, 1.648 mm and 0.857 mm respectively; the overall maximum error and average error are 2.362 mm and 2.221 mm respectively; the blue line is the weld point cloud obtained by using the plane intersection fitting method, and the errors of the corresponding weld points in the X, Y and Z directions are as shown in FIG. 6C. Figure 13 As shown in FIG. 6C, the maximum errors in the X, Y and Z directions are 0.161 mm, 1.091 mm and 0.533 mm respectively, and the average errors in the three directions are 0.079 mm, 0.456 mm and 0.251 mm respectively; the overall maximum error and average error are 1.197 mm and 0.555 mm respectively; in comparison, the positioning accuracy of the weld point cloud obtained by using the plane intersection fitting method is significantly better than that of the weld point cloud obtained by using the RANSAC linear fitting method, especially in terms of overall error.

[0211] In this embodiment, a plurality of experiments are designed to compare the performance parameters of the improved YOLOv11-seg model of the application with the original model and other models.

[0212] The performance comparison parameter table of the improved YOLOv11-seg model and the original model is shown in Table 1.

[0213] Table 1 Performance comparison parameter table of the improved YOLOv11-seg model and the original model

[0214]

[0215] As shown in Table 1, the complete model integrating the three modules of Meta-TSSA, Focal-Modulation and TDAS-Head has the best comprehensive performance compared with the original model based on integrating part of the modules or the original model. Compared with the basic model, the mAP75 and mAP50:95 are improved by 7.4% and 2.12% respectively, which are the highest values in all experiments. At the same time, the parameter amount is reduced by 13.05%, and the model size is reduced by 13.79%, which indicates that the model is more lightweight, and the inference speed (FPS) reaches 52.19, which is improved by 8.64%. Overall, the experimental data shows the effectiveness of the three modules introduced in the application, and their combination successfully balances the precision, speed and complexity of the model.

[0216] The performance comparison parameter table of the improved YOLOv11-seg model and other models in the prior art is shown in Table 2.

[0217] Table 2 Performance comparison parameter table of improved YOLOv11-seg model and other models

[0218]

[0219] As shown in Table 2, the improved YOLOv11-seg model (Improved-YOLOv11-seg) of the present application reaches the optimal value in all comparison algorithms in four indicators of key precision indicators mAP75, mAP50:95 and parameter quantity (Params) and model size (ModeSize) for measuring model complexity. Compared with the YOLOv11-seg model as the basis, the mAP75 and mAP50:95 precision of the improved YOLOv11-seg are significantly improved by 7.4% and 2.12% respectively, while the parameter quantity and model size are reduced by more than 13%, and the inference speed (FPS) is also improved from 48.04 to 52.19, having better inference speed. Compared with the Mamba-YOLO-seg model with more complex structure, the improved YOLOv11-seg realizes higher segmentation accuracy and 45% improvement of inference speed with less than half of the parameter quantity. Compared with YOLOv8-seg and YOLOv9t-seg algorithms, the improved YOLOv11-seg also has better segmentation accuracy under the premise of lighter model and less parameters. Although the inference speed is less than YOLOv12-seg, the mAP75 and mAP50:95 are increased by 8.45% and 3.47% respectively. In summary, the improved YOLOv11-seg model proposed in the present application has better comprehensive performance than other algorithms in the weld region segmentation task, and the FPS reaches 52.19, meeting the real-time segmentation demand in industry.

[0220] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A welding seam extraction method based on multi-dimension information fusion, characterized in that, Comprise the following steps: S1, acquire the Texture image and depth data of the weld through the structured light camera; S2, input the Texture image into the image recognition model to obtain the mask image of the weld, and the image recognition model is a YOLOv11-seg model; In the backbone network and neck network of the YOLOv11-seg model, a Meta-TSSA module is introduced to replace the pair similarity calculation of the original self-attention mechanism with the second-order statistics of Token features, which specifically comprises the following steps: Based on the maximum coding rate reduction, a feature interaction paradigm is constructed to realize feature transformation; A dynamic feature weighting mechanism is designed to realize adaptive feature selection; The context global information is integrated; In the backbone network of the YOLOv11-seg model, a Focal-Modulation module is introduced to replace the original SPPF module, which realizes the dynamic integration of multi-scale context information through hierarchical context encoding and gated aggregation mechanism, and generates output representation through the element-level interaction of the modulator and the query feature, which specifically comprises the following steps: Use depth separable convolution to construct hierarchical feature representation; Use gated aggregation mechanism to dynamically aggregate features at each level; Enhance feature representation through element-level modulation; A task dynamic alignment segmentation head is introduced to replace the original segmentation head, which improves the accuracy of weld region segmentation through task decomposition and feature alignment strategy, which specifically comprises the following steps: Extract features from the feature map output by the neck network through a shared convolution layer and reduce the dimension to obtain shared features; Decompose the shared features through a task decomposition module, that is, process the shared features through the following three branches: Classification branch: generate class decomposition features for target classification; Regression branch: generate bounding box regression decomposition features for target positioning; Prototype branch: generate prototype decomposition features for subsequent mask generation; Generate offset and mask according to the prototype decomposition features, and align the offset, mask and prototype decomposition features through a dynamic convolution alignment module to generate the final mask prediction result; Input the class decomposition features into the class probability generation module to generate the class probability of each pixel point to obtain the class prediction result; Input the regression decomposition features into the bounding box generation module to generate the bounding box prediction result; Scale and splice the mask prediction result, class prediction result and bounding box prediction result to obtain the final detection box and segmentation mask; S3, binarize the mask image to extract the pixel coordinates of the weld; S4, align the pixel coordinates of the weld with the depth data to generate the initial weld point cloud; S5, process the initial weld point cloud using a plane segmentation algorithm and a fitting algorithm to extract weld feature points; S6, perform linear fitting on the extracted weld feature points to obtain the accurate weld point cloud.

2. The weld seam extraction method based on multi-dimension information fusion according to claim 1, characterized in that, In step S2, the method for obtaining the weld mask image through the YOLOv11-seg model comprises: S21, the collected Texture image is input into the backbone network of the YOLOv11-seg model after pretreatment, and the hierarchical features of the Texture image are extracted; S22, different levels of features extracted by the backbone network are fused through the neck network of the YOLOv11-seg model, and information of different scales is considered; S23, the features fused by the neck network are input into the segmentation head of the YOLOv11-seg model, and a weld segmentation mask image is obtained.

3. The weld seam extraction method based on multi-dimension information fusion according to claim 1, characterized in that, In step S2, the feature change formula is as follows: ； wherein, denotes the second order statistics vector of the kth attention head, reflecting the weighted average of the sum of squares of the projected features, with dimension p x 1, p being the projection dimension; denotes the projection matrix of the kth attention head, used to project the high-dimensional features Z into a low-dimensional subspace; denotes the square operation on each element in the projected matrix ; denotes the grouping weight vector of the kth attention head, denotes the inner product with the all-one vector; Adaptive feature selection is realized by the following formula: ； wherein, denotes a diagonal matrix of the k-th attention head; denotes an operation that converts an input vector into a diagonal matrix; denotes taking the reciprocal of each element in a vector; A feature fusion formula is constructed to integrate the context global information, and the fusion formula is as follows: ； wherein, represents the first layer updated token feature vector; represents the first layer input token feature vector; K is the total number of attention heads.

4. The weld seam extraction method based on multi-dimension information fusion according to claim 2, characterized in that, In step S2, the hierarchical feature expression is as follows: ； wherein, is the level feature map, representing the feature map after hierarchical contextualization processing; is the level contextualization function, used for processing the feature map of the previous level; is the GeLU activation function, used for introducing nonlinearity; and DW-Conv is a deep separable convolution operation. The aggregation formula for dynamically aggregating features of each level is as follows: ； wherein, is an output feature representing an output feature obtained by aggregating features of each level by weighting; L is a number of levels of hierarchical processing; represents a gating weight of the level; represents an element-wise multiplication operation; The feature modulation formula is as follows: ； in, For the first A concise representation of a query token; This is a query projection function used to extract the feature vector at a specific location in the feature map. Linear projection is the query vector; For feature map location The eigenvector at that location; As a feature modulator, it aggregates features The result is obtained through lightweight processing and is used for query vectors. Perform feature enhancement. This is the input feature map.

5. The weld seam extraction method based on multi-dimension information fusion according to claim 1, characterized in that, In step S4, the method for generating the initial weld point cloud includes the following steps: S41, coordinate alignment: For each pixel point , if and , then the three-dimensional coordinates of the pixel point are calculated as: ； C is the pixel coordinate set of the weld extracted in step S3; is the depth value at the pixel point , obtained from the depth data; is the three-dimensional coordinate of the pixel point ; is the re-projection mapping vector, obtained from the intrinsic parameters of the structured light camera; If or , then let the three-dimensional coordinates of the point be ; S42, point cloud construction: All the pixel points The corresponding three-dimensional coordinate points are added to the point cloud set to obtain an initial weld point cloud.

6. The weld seam extraction method based on multi-dimension information fusion according to claim 1, characterized in that, In step S5, the method for extracting weld feature points includes the following steps: S51, plane segmentation and fitting: The initial weld point cloud is segmented using a region segmentation algorithm to obtain two plane regions; The two plane regions are fitted using a RANSAC plane fitting algorithm to obtain two plane equations: ； wherein , respectively denote the components of the normal vectors of the two planes in the direction of the axis; , respectively denote the intercepts of the two planes;​ The direction vector of the weld is calculated: ； wherein is the direction vector of the weld; S52, weld feature point extraction: Let Any value is zero, solve the plane equation, get a point on the weld line ; Computing the distance between a data point q in the weld point cloud to the intersection between the two planes : ； wherein representative point a projection point on the weld line, representative point a projection parameter to the weld line; Setting threshold If , the point is a weld feature, extract all weld feature points that satisfy the threshold condition .

7. The weld seam extraction method based on multi-dimension information fusion according to claim 1, characterized in that, In step S6, the extracted weld feature points are fitted using a RASCAN line fitting method, and an accurate weld point cloud is obtained through uniform interpolation.

8. The weld seam extraction method based on multi-dimension information fusion according to claim 1, characterized in that, In step S6, the extracted weld feature points are fitted using a plane intersection fitting method, and an accurate weld point cloud is obtained through uniform interpolation.

Images