A welding seam segmentation method based on fusion of visual and depth information
By combining a dual-stream coding network and a cross-modal semantic reasoning graph, the accuracy and robustness issues of existing technologies in weld seam detection in large-scale industrial welding scenarios are solved, achieving high-precision and high-robust weld seam segmentation results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF SCI & TECH
- Filing Date
- 2026-05-12
- Publication Date
- 2026-06-09
AI Technical Summary
Existing RGB-D network architectures are difficult to accurately depict the extremely slender topology and fine boundary features of weld seams in large-scale industrial welding scenarios. Furthermore, they are easily affected by writing marks on the workpiece surface, complex backgrounds, exposure differences, and interference from debris in complex environments, leading to noise interference during weld seam feature extraction and frequent false detections or missed detections.
A dual-stream coding network is used to extract features from RGB and depth images. The RGB coding features are recalibrated by generating a spatial attention map through the depth coding features. A cross-modal semantic reasoning graph is constructed in the decoding stage. The token nodes are updated using graph reasoning for weighted calculation to generate cross-modal enhanced fusion features. Finally, the weld prediction map is output.
It effectively reduces the false detection rate in complex industrial environments, enhances the ability to characterize fine weld boundaries and continuous topological structures, and achieves highly robust and high-precision weld segmentation to meet real-time industrial needs.
Smart Images
Figure CN122175976A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and deep learning technology, specifically to a weld seam segmentation method based on the fusion of visual and depth information. Background Technology
[0002] With the development of automated welding technology, semantic segmentation technology based on the fusion of visual and depth information (RGB-D) has been widely used in environmental perception and target detection in robotic welding systems. Currently, the core research in the field of RGB-D segmentation mainly focuses on designing robust multimodal fusion mechanisms, aiming to extract geometric feature information from the depth image to assist and enhance the conventional RGB image feature representation.
[0003] To achieve effective fusion of multimodal information, various representative network architectures and research paradigms have been proposed in the industry. For example, one type of research focuses on enhancing the complementarity between different modal features through bidirectional interaction mechanisms, such as using the interaction of spatial and channel dimensions to strengthen context modeling capabilities, or introducing adaptive depth thresholds combined with local channel attention to achieve complementary denoising of cross-modal features. Another type of research mainly addresses depth quality issues and feature level differences, by jointly utilizing the original depth map and the estimated depth map to enhance geometric priors, or by applying different fusion mechanisms to shallow detail information and deep semantic information in a progressive decoding framework. In addition, in recent years, researchers have extended fusion strategies to the Transformer architecture, directly introducing the depth map as a geometric prior into the self-attention mechanism to simplify the feature fusion process.
[0004] However, most existing RGB-D network architectures are general models designed for macroscopic targets (such as furniture, pedestrians, and vehicles) in typical indoor or outdoor scenes. When applied to actual large-scale industrial welding scenarios, the weld seam itself exhibits an extremely slender topological structure with very fine boundary features. Existing general models usually lack sensitivity to such microscopic fine structures and are difficult to accurately characterize the geometry of the weld seam. Secondly, real industrial sites often have serious interference factors, such as writing marks on the workpiece surface, complex and variable background conditions, exposure differences such as overexposure or underexposure, and interference from debris around the workpiece. Existing models lack targeted geometric feature constraints and anti-interference mechanisms, making it difficult to reduce noise interference in the weld seam feature extraction process in such complex environments, which can easily lead to false detections or missed detections. Therefore, there is an urgent need in this field to explore a dedicated RGB-D feature fusion architecture specifically designed for weld seam inspection tasks in complex industrial environments, so as to give full play to the guiding role of depth features on visual information and achieve highly robust and high-precision weld seam segmentation. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a weld seam segmentation method based on the fusion of visual and depth information, comprising: S1: acquiring an RGB image and a depth image of the component to be tested; S2: inputting the RGB image and the depth image into a two-stream coding network for processing. Level feature extraction, to obtain The RGB encoding features and depth encoding features corresponding to each encoding level S3: In each coding level, spatial attention maps are generated based on deep coding features and the corresponding RGB coding features are recalibrated. The recalibrated RGB coding features are fused with the corresponding deep coding features to generate cross-modal fusion features corresponding to each coding level. S4: RGB decoding features and deep decoding features for N decoding levels are generated via a dual-stream decoding network. The cross-modal fusion features generated at the last coding level are used as the target fusion features for the first decoding level. In each decoding level, global descriptors for the RGB decoding features, deep decoding features, and the target fusion features corresponding to the current level are extracted and a graph structure is constructed using the initialized token nodes. The token nodes are updated through graph inference, and the updated token nodes are used as gating factors to weight the target fusion features to generate the cross-modal enhanced fusion features for the current level. Except for the last decoding level, the cross-modal enhanced fusion features generated at the current level are upsampled and used as the target fusion features for the next decoding level. S5: Based on the cross-modal enhanced fusion features generated at the last decoding level, the weld prediction map is output.
[0006] Furthermore, the dual-stream coding network includes RGB coding branches and depth coding branches with mutually independent parameters; the dual-stream coding network uses a Res2Net backbone network to extract the N-level features at N different spatial scales; the dual-stream decoding network includes receptive field blocks with multi-branch structures to capture multi-scale contextual information under different receptive fields.
[0007] Furthermore, in S4, a graph structure containing token nodes is constructed. Updating the token nodes through graph inference specifically includes: extracting global descriptors of the corresponding RGB decoding features, depth decoding features, and target fusion features through global average pooling operations to generate corresponding RGB nodes, depth nodes, and fusion nodes; introducing the token node with an initial state of zero vector; establishing fully connected edges between the RGB nodes, depth nodes, and fusion nodes to construct a cross-modal subgraph that captures bidirectional dependencies between modalities; establishing aggregated edges connecting the RGB nodes, depth nodes, and fusion nodes to the token node to form the graph structure; and using a graph neural network to propagate information along the edges in the graph structure, aggregating the information of each neighboring node connected to the token node to obtain the updated token node.
[0008] Furthermore, in S4, the cross-modal enhanced fusion feature for the current level is generated through the following steps: the updated token node is processed by the Sigmoid activation function and used as a channel-level gating factor, which is then multiplied element-wise by the target fusion feature. The result of the multiplication operation is then added element-wise by the target fusion feature to generate the cross-modal enhanced fusion feature for the current decoding level. The calculation formula is:
[0009]
[0010] in, For the updated token node, This is the decoding level number. The cross-modal fusion features generated at the final coding level are used as the target fusion features for the first decoding level input. The cross-modal enhanced fusion features generated by the previous decoding level are upsampled and amplified, and then used as the target fusion features for the input of the remaining decoding levels. This represents the Sigmoid activation function. This indicates an element-wise multiplication operation by channel.
[0011] Furthermore, the S3 recalibrates the RGB encoding features, including: utilizing... Convolution operation on the RGB encoded features With deep encoding features By adjusting the channel dimensions, the adjusted RGB encoding features are obtained. With deep encoding features ; Regarding the conduct Convolution and Sigmoid activation function processing to generate spatial attention maps ; will the and Perform element-wise multiplication and then multiply the result by... Element-wise addition is performed to obtain the recalibrated RGB encoding features. The calculation formula is:
[0012]
[0013] in, This indicates an element-wise multiplication operation. This is the encoding level number.
[0014] Furthermore, generating cross-modal fusion features in S3 specifically includes: for the first coding level, the recalibrated RGB coding features... With adjusted deep coding features Passing through After convolution processing, channel splicing is performed, followed by... Convolutional processing generates the cross-modal fusion features corresponding to the first encoding layer. The calculation formula is:
[0015]
[0016] For the remaining encoding levels, the recalibrated RGB encoding features With adjusted deep coding features respectively After convolution, the cross-modal fusion features generated in the previous encoding layer are... After aligning the spatial resolution, channel stitching is performed, and then... Convolutional processing generates cross-modal fusion features at the current coding level. The calculation formula is:
[0017] ;
[0018] in, and They represent and Convolution operation, This indicates a channel splicing operation. It is the encoding level number greater than 1.
[0019] Furthermore, the method includes steps for optimizing and training the network, specifically: outputting RGB prediction maps and depth prediction maps respectively based on the RGB decoding features and depth decoding features generated by the final decoding layer; and constructing a single-branch joint loss function. It includes pixel-wise weighted cross-entropy loss. With structural level IoU loss The single-branch joint loss function is used to calculate the losses of the RGB prediction map, depth prediction map, and weld prediction map respectively, and the three are added together to form the total loss function. Synchronous optimization is performed; wherein, the pixel-level weighted cross-entropy loss medium pixel Importance weight According to pixels set of real labels of neighboring pixels Dynamic calculation of the degree of difference:
[0020]
[0021] in, For pixels The true label, The set of true labels for the neighboring pixels medium pixel The true label.
[0022] The joint optimization strategy alleviated the optimization difficulties caused by the extremely small proportion of weld seams in the entire image.
[0023] Furthermore, the dual-stream coding network and dual-stream decoding network are pre-trained using a cross-modal training dataset. The construction steps of the cross-modal training dataset include: S11: acquiring RGB images of a workpiece with weld seams, and phase-shifting fringe images of the left and right views, which are reflected from the surface of the workpiece and collected from different viewpoints; S12: performing phase extraction and unwrapping processing on the phase-shifting fringe images to obtain continuous absolute phases; S13: performing stereo matching between the left and right views based on the absolute phases, and reconstructing the three-dimensional point cloud of the workpiece with weld seams by combining the pre-acquired viewpoint calibration parameters; S14: extracting the Z-axis component from the three-dimensional point cloud, normalizing and linearly mapping it to the [0,255] interval to generate a depth image pixel-level aligned with the RGB images; S15: introducing preset industrial environmental interference factors into the shooting scene, dividing the samples into different pollution levels according to the number of introduced industrial environmental interference factors, and integrating the RGB images of each pollution level with the corresponding depth images to construct a cross-modal training dataset containing environmental features.
[0024] Furthermore, steps S12 and S13 specifically include: employing The phase-shifting algorithm processes the phase-shifting fringe image data to generate wrapped phase data, and the Gray encoding algorithm is used to expand the wrapped phase data to generate continuous absolute phase data. Based on the epipolar constraint condition, data points with the same absolute phase value are matched, and a three-dimensional spatial equation system is constructed in combination with the calibration parameter matrix. The three-dimensional spatial coordinates of the matched points are calculated to generate the three-dimensional point cloud data.
[0025] Furthermore, the industrial environmental interference factors include at least one of the following: writing marks on the workpiece surface, complex background conditions, different exposure intensities, and interference from debris around the workpiece; the pollution levels include: no pollution level, corresponding to a scenario that does not contain any of the aforementioned interference factors; light pollution level, corresponding to a scenario that contains only one of the aforementioned interference factors; moderate pollution level, corresponding to a scenario that contains two of the aforementioned interference factors; and heavy pollution level, corresponding to a scenario that contains three or more of the aforementioned interference factors.
[0026] Compared with the prior art, the present invention has the following advantages:
[0027] In the encoding stage, the physical properties of depth images to resist two-dimensional photometric interference are utilized to extract deep geometric priors as spatial attention maps and perform early recalibration of RGB features. This effectively filters background clutter and enhances the spatial saliency of the target before entering the deep decoding network, thereby reducing the false detection rate in complex industrial interference environments. In the decoding stage, by extracting global descriptors of each modality and introducing initialized learnable token nodes, a cross-modal semantic reasoning graph is constructed. This effectively establishes long-range dependencies across modalities, enabling the network to correct local feature expressions from a global perspective. This enhances the ability to characterize fine weld seam boundaries and continuous topological structures, effectively avoiding the fragmentation of prediction results in complex scenarios. Attached Figure Description
[0028] Figure 1 This is a flowchart illustrating the steps of a weld seam segmentation method based on the fusion of visual and depth information provided in an embodiment of the present invention.
[0029] Figure 2 This is a schematic diagram of the overall architecture of the dual-stream coding network and dual-stream decoding network provided in the embodiments of the present invention;
[0030] Figure 3 This is a schematic diagram of the internal structure of the Depth-Guided Cross-Modal Attention Module (DGCA) provided in an embodiment of the present invention;
[0031] Figure 4 This is a schematic diagram of the cross-modal enhancement module based on graph structure in an embodiment of the present invention;
[0032] Figure 5 This is a flowchart illustrating the steps involved in constructing a cross-modal training dataset according to an embodiment of the present invention.
[0033] Figure 6 Comparison of weld seam segmentation effects between embodiments of the present invention and other existing methods in complex industrial environments. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.
[0035] refer to Figures 1-6 As shown in the figure, this embodiment provides a weld seam segmentation method based on the fusion of visual and depth information. The two-stream coding network and the two-stream decoding network involved are pre-trained using a cross-modal training dataset. To enable the model to have robust segmentation capabilities in complex industrial scenarios, the construction steps of the cross-modal training dataset specifically include:
[0036] S11: Acquire an RGB image of the workpiece with weld seam, and a left-view phase-shifted fringe image and a right-view phase-shifted fringe image that are reflected from the surface of the workpiece at different viewpoints.
[0037] This embodiment uses an automated welding platform that integrates hand and eye. It uses a binocular structured light camera to acquire RGB images of the workpiece with weld seams and controls a projector to project stripe images onto the workpiece surface. The binocular camera simultaneously acquires phase-shifted stripe images of the left and right views reflected from the workpiece surface. To simulate actual large components, SLA 3D printing technology can be used to prepare diverse workpiece models.
[0038] S12: Perform phase extraction and unwrapping processing on the phase-shifted fringe image to obtain a continuous absolute phase.
[0039] The phase-shifted fringe image data is processed using an N-step phase-shift algorithm to generate wrap-around phase data. The corresponding intensity distribution expression recorded by the binocular camera is:
[0040]
[0041] in, This represents the average intensity of the striped image. Indicates modulation intensity. Indicates the package phase. This represents the number of phase shift steps. The enclosed phase value is obtained by solving for it.
[0042]
[0043] Since the calculated phase is contained within a specific interval, the Gray encoding algorithm is further used to expand the contained phase data to generate continuous absolute phase data. The calculation formula is as follows:
[0044]
[0045] in, The encoded value is determined by the distribution of pixels within the black and white stripe interval. This is the generated continuous absolute phase data.
[0046] S13: Based on the absolute phase, perform stereo matching between the left and right views, and reconstruct the three-dimensional point cloud of the workpiece with weld seam by combining the pre-acquired viewpoint calibration parameters.
[0047] Based on epipolar constraints, data points with the same absolute phase value are matched. A three-dimensional spatial equation system is constructed using the calibration parameter matrix, and the three-dimensional spatial coordinates of the matched points are calculated to generate the three-dimensional point cloud data. Let the coordinates of the three-dimensional points in the world coordinate system be... Its homogeneous image coordinates in the left and right cameras are respectively and Establish the association based on the geometric principles of binocular imaging:
[0048]
[0049]
[0050] in, and These represent the depth values of the 3D point in the left and right camera coordinate systems, respectively. and Let represent the parameter matrices (i.e., viewpoint calibration parameters) of the left and right cameras, respectively. Simplifying the above system of equations will allow us to recover the world coordinates of the 3D points.
[0051] S14: Extract the Z-axis component from the 3D point cloud, normalize and linearly map it to the [0, 255] interval, generate a depth image pixel-aligned with the RGB image, and then extract the Z-axis component from the 3D point cloud. After the axis components are normalized, an aligned depth map can be generated, providing a high-quality data source for the subsequent two-stream segmentation network.
[0052] S15: Introduce preset industrial environmental interference factors into the shooting scene. Divide the samples into different contamination levels based on the number of introduced industrial environmental interference factors. Integrate the RGB images of each contamination level with the corresponding depth images to construct a cross-modal training dataset containing environmental features. The industrial environmental interference factors include at least one of the following: writing marks on the workpiece surface, complex background conditions, different exposure intensities, and clutter interference around the workpiece. The specific contamination levels include:
[0053] No pollution level, corresponding to scenarios that do not contain any of the aforementioned interfering factors;
[0054] A light pollution level corresponds to a scenario containing only one of the aforementioned interfering factors;
[0055] Moderate pollution level, corresponding to a scenario containing two of the aforementioned interfering factors;
[0056] Severe pollution level corresponds to scenarios containing three or more of the aforementioned interfering factors.
[0057] After completing the construction of the cross-modal training dataset and model pre-training, the weld seam segmentation method based on the fusion of visual and depth information of this invention specifically includes the following steps in the inference stage:
[0058] S1: Acquire the RGB and depth images of the component to be tested, and scale the original images to [specific format] through data preprocessing. A fixed resolution.
[0059] S2: Input the RGB image and depth image into a dual-stream coding network for N-level feature extraction to obtain RGB coding features and depth coding features corresponding to N coding levels, where N is an integer greater than 1.
[0060] refer to Figure 2 As shown, the dual-stream coding network includes an RGB coding branch and a depth coding branch with non-shared parameters; the dual-stream coding network uses a Res2Net backbone network to extract the N-level features at N different spatial scales. In this embodiment, it is set... In the At each scale, the extracted RGB encoded features and deep encoded features are respectively represented as follows: and As the network deepens, the spatial resolution of the feature maps gradually decreases, providing a rich hierarchical feature base for subsequent cross-modal fusion.
[0061] S3: In each coding level, spatial attention maps are generated based on deep coding features and the corresponding RGB coding features are recalibrated. The recalibrated RGB coding features are then fused with the corresponding deep coding features to generate cross-modal fusion features for each coding level.
[0062] For recalibration and fusion at each coding level, such as Figure 3 As shown, this embodiment introduces a depth-guided cross-modal attention module (DGCA), the specific implementation steps of which include: utilizing... Convolution operation on the RGB encoded features With deep encoding features By adjusting the channel dimensions, the adjusted RGB encoding features are obtained. With deep encoding features :
[0063]
[0064] Subsequently, regarding the aforementioned conduct Convolution and Sigmoid activation function processing to generate spatial attention maps :
[0065]
[0066] The and Perform element-wise multiplication and then multiply the result by... Element-wise addition is performed to obtain the recalibrated RGB encoding features. The calculation formula is:
[0067]
[0068] in, This indicates an element-wise multiplication operation. For the encoding level number, when generating cross-modal fusion features: for the first encoding level (i.e. The recalibrated RGB encoding features With adjusted deep coding features Passing through After convolution processing, channel splicing is performed, followed by... Convolutional processing generates the cross-modal fusion features corresponding to the first encoding layer. The calculation formula is:
[0069]
[0070] For the remaining coding levels (i.e.) The recalibrated RGB encoding features With adjusted deep coding features respectively After convolution, the cross-modal fusion features generated in the previous encoding layer are... After max pooling operations (such as size) After aligning the spatial resolution with a step size of 2, perform channel stitching with the current level features; then... Convolutional processing generates cross-modal fusion features at the current coding level. The calculation formula is:
[0071]
[0072] in, This indicates a channel splicing operation.
[0073] S4: Through a dual-stream decoding network, RGB decoding features and depth decoding features for N decoding levels are generated. The cross-modal fusion feature generated at the final encoding level is used as the target fusion feature for the first decoding level. In each decoding level, the global descriptors of the RGB decoding feature, depth decoding feature, and target fusion feature corresponding to the current level are extracted and combined with the initialized token nodes to construct a graph structure. The token nodes are updated through graph reasoning, and the updated token nodes are used as gating factors to perform weighted calculations on the target fusion feature to generate the cross-modal enhanced fusion feature for the current level. Except for the final decoding level, the cross-modal enhanced fusion feature generated at the current level is upsampled and used as the target fusion feature for the next decoding level.
[0074] To capture global context information under different receptive fields, the dual-stream decoding network includes receptive field blocks (RFBs) with a multi-branch structure. This embodiment adopts a dual-decoding structure following the U-Net design philosophy.
[0075] At each decoding level In this process, by upsampling and combining the skip connection features of the corresponding layer of the encoding network, the RGB decoding features of the current layer are generated respectively. With depth decoding features ;
[0076] The decoding network also consists of 5 decoding layers (i.e. To overcome the limitations of the local receptive field in standard convolution, this embodiment introduces a graph-based cross-modal enhancement module (GCE) in each decoding layer. This constructs a graph structure containing token nodes, specifically including: extracting the corresponding RGB decoding features through global average pooling. Deep decoding features And the global descriptor of the target fusion features, generating the corresponding RGB nodes. Deep nodes With fusion node :
[0077] ;
[0078] ;
[0079] ;
[0080] Introducing the token node with an initial state of zero vector The set of initial nodes thus formed is defined as Establish fully connected edges between the RGB node, depth node, and blend node. To construct a cross-modal subgraph that captures bidirectional dependencies between modalities:
[0081] ;
[0082] Establish aggregate edges connecting the RGB node, depth node, and fusion node to the token node to form the graph structure. After constructing the graph structure, a graph neural network (such as GAT or a single-layer Graph Transformer) is used for graph reasoning to update the token node. Then, a weighted calculation is performed on the target fusion features, specifically including: updating the token node... After processing with the Sigmoid activation function, the gate factor, serving as the channel-dimensional gating factor, is spatially expanded to match the size of the target fusion feature. This gating factor is then multiplied element-wise by channel with the target fusion feature. The result of this multiplication is then added element-wise by channel to the target fusion feature (i.e., residual enhancement), generating the cross-modal enhanced fusion feature for the current decoding level. The calculation formula is:
[0083]
[0084] in, This is the decoding level sequence number; for ease of writing, This represents the target fusion features after upsampling, as shown in the above logic, in the first decoding level ( The input target fusion feature is the cross-modal fusion feature generated at the final coding level. (In this embodiment) In the remaining decoding levels ( The input target fusion feature is the cross-modal enhanced fusion feature generated by the previous decoding layer. Feature maps with increased spatial resolution after bilinear interpolation upsampling. This represents the Sigmoid activation function. This indicates an element-wise multiplication operation by channel.
[0085] S5: Based on the cross-modal enhanced fusion features generated by the final decoding layer, output the weld prediction map.
[0086] Through the aforementioned hierarchical cross-modal enhancement and feature weighting, the network can fully mine global and local information, and finally output a high-precision weld prediction map through the final-level decoding features.
[0087] Since weld seams occupy a very small proportion of the entire image or workpiece area in real industrial scenarios, conventional cross-entropy loss can easily lead to extreme imbalance between positive and negative samples. Therefore, this embodiment adopts a joint loss function. The RGB prediction map, depth prediction map, and weld prediction map (i.e., the RGB-D fusion prediction map output at the end) are optimized synchronously.
[0088] The joint loss function includes pixel-wise weighted cross-entropy loss. With structural level IoU loss The specific calculation mechanism is as follows:
[0089] (1) Pixel-wise weighted cross-entropy loss The main focus is on pixel-level prediction optimization of regions of interest, wherein the pixel-level weighted cross-entropy loss... medium pixel Importance weight According to pixels set of real labels of neighboring pixels Dynamic calculation of the degree of difference:
[0090]
[0091] in, For pixels The true label, The set of true labels for the neighboring pixels medium pixel The real label, when pixel When the difference from neighboring pixels is large (i.e., at the edge of the weld), Larger differences are assigned higher weights; conversely, smaller differences are assigned lower weights.
[0092] To ensure the model retains basic gradient backpropagation in smooth regions and avoids complete failure to optimize, this embodiment is based on the aforementioned importance weights. Introducing hyperparameters The complete pixel-wise weighted cross-entropy loss function is constructed as follows:
[0093]
[0094] in, This represents the total number of pixels in the image. Indicates the first The predicted value of each pixel.
[0095] (2) Structural-level IoU loss This loss focuses more on the direct optimization of the overall target structure, rather than the accuracy of individual pixel predictions. It ensures the complete prediction of the slender topology of the weld and has natural robustness to the problem of imbalanced samples. The calculation formula is as follows:
[0096]
[0097] Therefore, the joint optimization loss function for a single prediction branch can be expressed as:
[0098]
[0099] (3) Synchronous optimization of the total loss function:
[0100] Finally, the joint loss functions corresponding to the RGB prediction map, depth prediction map, and weld prediction map are added together to form the total loss function. The calculation formula is:
[0101]
[0102] in, , and These represent weld prediction maps for the RGB branch, depth branch, and RGB-D fusion branch, respectively. A true diagram showing the weld seam. , and These represent the joint loss functions for the corresponding RGB, depth, and RGB-D fusion decoders, respectively. Through this synchronous optimization mechanism, the network can more robustly characterize the structural features of the weld.
[0103] To verify the effectiveness and superiority of the weld seam segmentation method based on the fusion of vision and depth information provided in this embodiment of the invention, the following detailed description is provided in conjunction with specific experimental data and comparison results.
[0104] Experimental Dataset and Implementation Details: The experiments in this embodiment of the invention were conducted on a constructed RGB-D-WSD cross-modal training dataset. The dataset contains 4800 pairs of welding workpiece samples covering different levels of environmental pollution (such as writing marks, complex backgrounds, different exposure intensities, and interference from debris), and is divided into training and test sets in an 8:2 ratio. This embodiment is based on a deep learning framework and is trained end-to-end on an NVIDIA RTX 4090 GPU. The model uses the Adam optimizer, with an initial learning rate of 1×10⁻³, 50 training epochs, and a batch size of 4. For data augmentation, the input images are uniformly scaled to 800×800 and then subjected to random rotation, flipping, random cropping, and color perturbation operations.
[0105] To comprehensively evaluate the performance of the model in this invention, mean absolute error (MAE) and F-measure were selected. ), structural similarity index ( Enhanced alignment metrics Five evaluation indicators, including the intersection-over-union ratio (IoU).
[0106] Overall performance and beneficial effects analysis
[0107] (1) Quantitative performance comparison: The method provided in the embodiments of the present invention is compared with the existing general RGB-D segmentation methods. The results are shown in Table 1 below. (Note: UCNet, DANet, PGAR, SPNet, HIDANet and MAGNet in the table are all existing segmentation models in the field.)
[0108] Table 1. Comparison results of the embodiments of the present invention and existing methods on the RGB-D-WSD dataset.
[0109]
[0110] As shown in Table 1, the method of the present invention achieved the best performance in most evaluation indicators, especially the IoU index, which reached 0.789, while reducing MAE to 0.650, indicating that the present invention has significant advantages in characterizing the fine-grained topology of welds.
[0111] (2) Real-time performance and inference efficiency: In industrial applications, the welding guidance of robots needs to meet real-time requirements. As shown in Table 2, this embodiment analyzes the computational complexity and inference delay of each method.
[0112] Table 2 Comparison of computational complexity and inference speed between the present invention and existing methods
[0113]
[0114] Note: In Table 2, FLOPs represents the number of floating-point operations, used to measure computational load; FPS represents the number of frames transmitted per second, used to measure inference speed.
[0115] Although the number of model parameters and computational cost have increased in order to support high-capacity graph inference mechanisms and cross-modal attention modules, the real-time inference speed of 24 FPS can still be achieved on an RTX 4090 GPU, which is within the acceptable latency range for actual welding inspection, achieving a good balance between segmentation accuracy, welding quality assurance and engineering practicality.
[0116] (3) Qualitative visual effect verification: as shown in the appendix Figure 6 As shown, under conditions of strong environmental interference (such as...) Figure 6 (The red box in the middle indicates overexposure / underexposure or complex background interference). Existing comparison methods are prone to serious false detections. However, this invention introduces depth information as a geometric prior into the model, which effectively suppresses the over-reliance on texture information and can stably generate continuous, clean and structurally consistent segmentation results, proving the robustness of this invention in complex industrial scenarios.
[0117] Ablation experiments of modules in the embodiments
[0118] To further verify the scientific validity and necessity of the various features in the claims of this invention, the following ablation experiments were conducted:
[0119] (1) The necessity of modal fusion: As shown in Table 3, relying solely on a single RGB modality or a single depth modality will result in a performance bottleneck due to limited information expression. This invention achieves a stable improvement in all evaluation indicators after the two modalities are fused together (MAE decreased by 8.8% compared to the RGB baseline), proving the key role of combining visual texture and geometric structure information in eliminating weld ambiguity.
[0120] Table 3 Comparison of experimental results between single-mode and two-stream fusion
[0121]
[0122] (2) Necessity of Attention Recalibration and Graph Inference Mechanism: This embodiment modularly verifies two core steps: Step A: Generate spatial attention maps based on deep coding features and recalibrate the corresponding RGB coding features (referred to as the Deep Guided Cross-Modal Attention Module, DGCA in the experiment); Step B: Construct a graph structure by combining token nodes, update token nodes through graph inference, and generate cross-modal enhanced fusion features (referred to as the Graph Structure-Based Cross-Modal Enhancement Module, GCE in the experiment); The results are shown in Table 4, indicating that neither Step A nor Step B alone can achieve optimal performance. Combining the two completely yields the lowest MAE (0.650) and the highest IoU (0.789). This demonstrates that the combination of noise suppression in the encoding stage (feature purification in Step A) and structured inference in the decoding stage (recovering fine-grained details in Step B) has a significant hierarchical synergistic effect.
[0123] Table 4 shows the ablation experimental results of the contributions of DGCA and GCE.
[0124]
[0125] In summary, the weld seam segmentation method based on the fusion of visual and depth information provided by this invention effectively overcomes the impact of complex interference in the industrial environment on weld seam detection by designing a network architecture and fusion strategy, achieving high precision, high robustness, and weld seam segmentation results that meet real-time industrial requirements.
[0126] The above description is only the best specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the protection scope of the present invention.
Claims
1. A weld seam segmentation method based on the fusion of visual and depth information, characterized in that, include: S1: Acquire the RGB image and depth image of the component under test; S2: Input the RGB image and depth image into a dual-stream coding network respectively. Level feature extraction, to obtain The RGB encoding features and depth encoding features corresponding to each encoding level It is an integer greater than 1; S3: In each coding level, spatial attention maps are generated based on deep coding features and the corresponding RGB coding features are recalibrated. The recalibrated RGB coding features are then fused with the corresponding deep coding features to generate cross-modal fusion features for each coding level. S4: Through the dual-stream decoding network, N decoding levels of RGB decoding features and depth decoding features are generated accordingly; The cross-modal fusion features generated at the final coding level are used as the target fusion features for the first decoding level. In each decoding level, the global descriptors of the RGB decoding features, depth decoding features, and target fusion features corresponding to the current level are extracted respectively, and a graph structure is constructed by combining the initialized token nodes; the token nodes are updated through graph reasoning, and the updated token nodes are used as gating factors to perform weighted calculations on the target fusion features to generate the cross-modal enhanced fusion features of the current level; In addition to the final decoding level, the cross-modal enhanced fusion features generated at the current level are upsampled and used as the target fusion features for the next decoding level. S5: Based on the cross-modal enhanced fusion features generated by the final decoding layer, output the weld prediction map.
2. The weld seam segmentation method based on vision and depth information fusion according to claim 1, characterized in that, The dual-stream coding network includes RGB coding branches and depth coding branches with mutually independent parameters; The dual-stream coding network uses a Res2Net backbone network to extract the N-level features at N different spatial scales; The dual-stream decoding network includes receptive field blocks with a multi-branch structure to capture multi-scale contextual information under different receptive fields.
3. The weld seam segmentation method based on vision and depth information fusion according to claim 1, characterized in that, In S4, constructing a graph structure containing token nodes and updating token nodes through graph reasoning specifically includes: Global average pooling is used to extract global descriptors for the corresponding RGB decoding features, depth decoding features, and target fusion features, generating corresponding RGB nodes, depth nodes, and fusion nodes. The token node with an initial state of zero vector is introduced. Fully connected edges are established between the RGB nodes, depth nodes, and fusion nodes to construct a cross-modal subgraph that captures bidirectional dependencies between modalities. Aggregate edges are established to connect the RGB nodes, depth nodes, and fusion nodes to the token node, forming the graph structure. A graph neural network is used to propagate information along the edges of the graph structure, aggregating the information of each neighboring node connected to the token node to obtain an updated token node.
4. The weld seam segmentation method based on vision and depth information fusion according to claim 1, characterized in that, In S4, the cross-modal enhanced fusion features of the current level are generated through the following steps: The updated token node, after being processed by the Sigmoid activation function, is used as a channel-level gating factor and multiplied element-wise by the target fusion feature. The result of this multiplication is then added element-wise by channel to the target fusion feature to generate the cross-modal enhanced fusion feature for the current decoding level. The calculation formula is: in, For the updated token node, This is the decoding level number. The cross-modal fusion features generated at the final coding level are used as the target fusion features for the first decoding level input. The cross-modal enhanced fusion features generated by the previous decoding level are upsampled and amplified, and then used as the target fusion features for the input of the remaining decoding levels. This represents the Sigmoid activation function. This indicates an element-wise multiplication operation by channel.
5. The weld seam segmentation method based on vision and depth information fusion according to claim 1, characterized in that, The recalibrated RGB encoding features in S3 include: Use separately Convolution operation on the RGB encoded features With deep encoding features By adjusting the channel dimensions, the adjusted RGB encoding features are obtained. With deep encoding features ; Regarding the conduct Convolution and Sigmoid activation function processing to generate spatial attention maps ; The and Perform element-wise multiplication and then multiply the result by... Element-wise addition is performed to obtain the recalibrated RGB encoding features. The calculation formula is: in, This indicates an element-wise multiplication operation. This is the encoding level number.
6. The weld seam segmentation method based on vision and depth information fusion according to claim 5, characterized in that, The generation of cross-modal fusion features in S3 specifically includes: for the first coding level, the recalibrated RGB coding features... With adjusted deep coding features Passing through After convolution processing, channel splicing is performed, followed by... Convolutional processing generates the cross-modal fusion features corresponding to the first encoding layer. The calculation formula is: For the remaining encoding levels, the recalibrated RGB encoding features With adjusted deep coding features respectively After convolution, the cross-modal fusion features generated in the previous encoding layer are... After aligning the spatial resolution, channel stitching is performed, and then... Convolutional processing generates cross-modal fusion features at the current coding level. The calculation formula is: ； in, and They represent and Convolution operation, This indicates a channel splicing operation. This is the encoding level number.
7. The weld seam segmentation method based on vision and depth information fusion according to claim 1, characterized in that, It also includes steps for optimizing and training the network, specifically including: Based on the RGB decoding features and depth decoding features generated by the final decoding layer, RGB prediction maps and depth prediction maps are output respectively. Construct a single-branch joint loss function It includes pixel-wise weighted cross-entropy loss. With structural level IoU loss The single-branch joint loss function is used to calculate the losses of the RGB prediction map, depth prediction map, and weld prediction map respectively, and the three are added together to form the total loss function. Perform synchronous optimization; Among them, the pixel-level weighted cross-entropy loss medium pixel Importance weight According to pixels set of real labels of neighboring pixels Dynamic calculation of the degree of difference: in, For pixels The true label, The set of true labels for the neighboring pixels medium pixel The true label.
8. The weld seam segmentation method based on vision and depth information fusion according to claim 1, characterized in that, The dual-stream coding network and dual-stream decoding network are pre-trained using a cross-modal training dataset. The steps for constructing the cross-modal training dataset include: S11: Acquire an RGB image of the workpiece with weld seam, and a left-view phase shift texture image and a right-view phase shift texture image reflected from the surface of the workpiece at different viewpoints; S12: Perform phase extraction and unwrapping processing on the phase-shifted fringe image to obtain a continuous absolute phase; S13: Based on the absolute phase, perform stereo matching between the left and right views, and reconstruct the three-dimensional point cloud of the workpiece with weld seam by combining the pre-acquired viewpoint calibration parameters. S14: Extract the Z-axis component from the three-dimensional point cloud, normalize and linearly map it to the [0,255] interval to generate a depth image that is pixel-level aligned with the RGB image; S15: Introduce preset industrial environmental interference factors into the shooting scene, divide the samples into different pollution levels according to the number of industrial environmental interference factors introduced, integrate the RGB images of each pollution level with the corresponding depth images, and construct a cross-modal training dataset containing environmental features.
9. The weld seam segmentation method based on vision and depth information fusion according to claim 8, characterized in that, Steps S12 and S13 specifically include: use The phase-shifting algorithm processes the phase-shifting stripe image data to generate wrap-around phase data, and the Gray encoding algorithm is used to unfold the wrap-around phase data to generate continuous absolute phase data; Based on the epipolar constraint condition, data points with the same absolute phase value are matched, and a three-dimensional spatial equation system is constructed in combination with the calibration parameter matrix. The three-dimensional spatial coordinates of the matched points are calculated to generate the three-dimensional point cloud data.
10. The weld seam segmentation method based on vision and depth information fusion according to claim 8, characterized in that, The industrial environmental interference factors include at least one of the following: writing marks on the workpiece surface, complex background conditions, different exposure intensities, and interference from debris around the workpiece. The pollution levels include: No pollution level, corresponding to scenarios that do not contain any of the aforementioned interfering factors; A light pollution level corresponds to a scenario containing only one of the aforementioned interfering factors; Moderate pollution level, corresponding to a scenario containing two of the aforementioned interfering factors; Severe pollution level corresponds to scenarios containing three or more of the aforementioned interfering factors.