Multi-layer multi-pass weld feature recognition method and system based on resnet-cgan

Abstract

The present application belongs to the technical field of weld defect image recognition, and particularly relates to a multi-layer multi-pass weld feature recognition method and system based on ResNet-CGAN, which comprises: collecting image data of multi-layer multi-pass welding; inputting the image data into a model of a pre-trained fusion residual network and conditional generative adversarial network, the model comprising a generator network and a discriminator network; the generator network is constructed based on an encoder-decoder structure of the residual network and is enhanced by fusing a color attention mechanism to generate a high-quality feature map; the discriminator network discriminates between the high-quality feature map and a real feature map label and optimizes the training of the generator network; and weld centerline and weld feature point information are extracted from the high-quality feature map output by the optimized generator network. The present application realizes high-precision and high-real-time weld information extraction and provides reliable data support for robot path planning.

Description

Technical Field

[0001] This invention belongs to the field of weld defect image recognition technology, specifically involving a multi-layer, multi-pass weld feature recognition method and system based on ResNet-CGAN. Background Technology

[0002] Welding is a key process in modern manufacturing, especially when joining medium-thick plates with a thickness exceeding 4mm, where multi-layer, multi-pass welding is indispensable. In multi-layer, multi-pass welding, "multi-layer" refers to the fusion of two or more weld layers to complete the entire weld, while "multi-pass" refers to the fusion of two or more weld passes to complete the entire weld. As the number of layers increases, the number of passes per layer also increases, and the weld centerline and weld feature points also change. However, traditional welding methods (such as manual or semi-automatic welding) are inefficient and labor-intensive, making them difficult to adapt to the demands of automated production. Robotic automated welding has emerged to address this need, with accurate and real-time acquisition of weld information being the core of achieving intelligent welding.

[0003] The research and application of intelligent and efficient welding of medium and heavy plates by combining laser vision sensors and deep learning methods has become the key to improving the quality of multi-layer and multi-pass welding processes.

[0004] To address the problem of blurred groove features caused by molten pool buildup, arc light interference, and thermal deformation in multi-layer, multi-pass welding, the following technical challenges are mainly resolved: 1. Blurred bevel information features and difficulty in positioning: As the number of weld layers increases, molten pool accumulation and thermal deformation lead to blurred bevel boundaries, making it difficult for traditional machine vision methods to reliably extract high-precision weld information; 2. Severe noise interference during welding: Arc light, spatter, and background noise significantly reduce the signal-to-noise ratio of the image during welding, affecting the detection of weld feature points; 3. Decreased feature consistency: In the case of weld bead overlay, the weld centerline and weld feature points change dynamically, resulting in large error fluctuations when traditional models extract features; 4. Insufficient real-time performance: Existing methods are difficult to meet the industrial requirements of real-time feature extraction (>30 FPS) in high-speed welding scenarios.

[0005] The most crucial step in intelligent multi-layer, multi-pass welding is detecting and acquiring basic weld information, including weld position, height, width, and initial point information. Traditional machine vision methods can effectively perform initial weld positioning, but the arc light during welding significantly impacts their positioning accuracy, ultimately leading to increased errors and a severe decline in weld quality. With the rapid development of industrial automation, laser vision sensors have been widely applied in robotic automated welding. Compared to traditional detection methods, these sensors possess the ability to quickly capture and process images, providing higher measurement accuracy and resolution. Furthermore, laser vision sensors can be well integrated with robotic automation systems, providing real-time monitoring and feedback. In multi-layer, multi-pass welding, laser vision sensors use lasers as the active light source, combined with CCD vision sensors, to extract weld depth information and accurately reconstruct weld shape. This characteristic makes them particularly outstanding in acquiring images of bevel welds.

[0006] Deep learning technology has made significant progress in computer vision and image processing, particularly in object detection and feature extraction, which has greatly improved the effectiveness of related research. For the detection of feature points in multi-layer, multi-pass weld bevels, deep learning models can automatically extract the centerline of laser lines from welding images, effectively identifying and extracting weld feature points. This method not only improves the accuracy and efficiency of weld feature point detection but also provides new insights for welding trajectory planning. By analyzing weld features and receiving real-time feedback, deep learning technology can achieve more precise welding path design, optimize the welding process, and thus effectively improve welding quality and production efficiency.

[0007] Traditional CNN models suffer from gradient vanishing in deep networks, leading to a decrease in feature map resolution, loss of deep network details, and an inability to recover blurred bevel boundaries. Existing methods lack attention mechanisms designed for laser color, resulting in insufficient color sensitivity and weak background noise suppression. The generated weld feature points are loosely distributed, with large geometric matching errors with real welds and a lack of geometric consistency constraints. Traditional models suffer from low frame rates due to parameter redundancy, failing to meet the requirements of high-speed welding. Summary of the Invention

[0008] Purpose of the Invention: The purpose of this invention is to solve the problem of multi-layer, multi-pass weld seam and weld feature point reconstruction and recognition under the above-mentioned complex conditions. It proposes a multi-layer, multi-pass weld seam feature recognition method and system that integrates ResNet and Conditional Generative Adversarial Network (CGAN) to solve the problem of fuzzy bevel feature extraction. A color attention mechanism is designed to enhance the model's sensitivity to the weld centerline (red / green / blue) and suppress background interference. An auxiliary centroid loss function is introduced to constrain the spatial distribution of weld feature points through connected component analysis, reducing positioning error fluctuations and achieving high-precision, high-real-time weld information extraction, providing reliable data support for robot path planning.

[0009] Technical solution: The multi-layer, multi-pass weld feature recognition method based on ResNet-CGAN described in this invention includes the following steps: Acquire image data of multi-layer, multi-pass welding; The image data is input into a pre-trained model that fuses a residual network and a conditional generative adversarial network. The model includes a generator network and a discriminator network. The generator network is constructed based on the encoder-decoder structure of the residual network and incorporates a color attention mechanism to enhance the weld centerline and weld feature point regions of a preset color channel, thereby generating a high-quality feature map. The discriminator network distinguishes between the high-quality feature map and the real feature map annotations, and optimizes the generator network through a combined loss function that combines conditional generative adversarial network loss, pixel-based content loss, and geometric constraint loss based on the spatial distribution of weld feature points. Weld centerline and weld feature point information are extracted from the high-quality feature map output by the optimized generator network.

[0010] The encoder progressively extracts multi-scale features from the image, and the decoder combines the information from the encoder (through skip connections) to restore spatial resolution and detail. Crucially, the generator network (G) incorporates a color attention mechanism. The generator network (G) learns to map from the input low-quality weld image to a high-quality feature map, where the weld centerline and weld feature points are significantly enhanced and sharpened in the output high-quality feature map.

[0011] During the training phase, the discriminator network's task is to distinguish between "fake" high-quality feature maps generated by the generator network and real feature map annotations (usually manually precisely annotated or ideal feature maps obtained through other high-precision methods). Trained using a Conditional Generative Adversarial Network (CGAN) framework, the generator network (G) strives to generate increasingly realistic feature maps to "deceive" the discriminator network (D), while the discriminator network continuously improves its discrimination ability. This adversarial process drives the generator network to learn to generate feature maps that are visually and distributionally highly similar to real annotations.

[0012] To accurately guide the training of the generator network, this invention employs a combined loss function, which includes not only the standard CGAN loss (used to ensure the realism of the generated images), but also: pixel-based content loss to ensure that the generated images are similar to the real annotations at the pixel level, and geometric constraint loss based on the spatial distribution of weld feature points, which aims to directly constrain the spatial position of the generated feature points to make them more consistent with the geometry of the actual weld.

[0013] After the model training is completed, the weld seam images acquired in real time are input into the optimized generator network (G). The required weld seam centerline and weld seam feature points (e.g., the coordinates of the bevel apex and bottom point) are extracted from the high-quality feature map output by the generator network. The extracted information has high accuracy and high robustness.

[0014] To further improve the above technical solution, the generator network includes residual blocks, encoders, and decoders. The residual blocks transmit information between the encoder and decoder paths through skip connections. Both the encoder and decoder are embedded with the color attention mechanism.

[0015] The ResNet architecture utilizes residual block design, allowing information to be directly passed across layers through skip connections. This effectively alleviates the gradient vanishing problem in deep network training, enabling the network to be deeper. At the same time, it can effectively pass shallow detail information in the encoder path (which is important for recovering blurred boundaries) to the decoder path, improving the detail recovery capability of feature maps.

[0016] Furthermore, the color attention mechanism enhances the response to a preset color by assigning learnable weights to different color channels and utilizing local max pooling operations.

[0017] Furthermore, the preset color corresponding to the weld centerline is red; the weld feature points include weld feature points at the bottom of the weld bead and weld feature points at the top of the weld bead, with preset colors of green and blue, respectively.

[0018] Furthermore, the assignment of learnable weights to different color channels includes generating a learnable scaling factor as a weight independently for each color channel; the local max pooling operation includes using sliding window max pooling to extract the highest response in a local region; and the color attention mechanism includes multiplying the output of the local max pooling operation by the learnable weight of the corresponding channel, processing it through a sigmoid activation function to obtain the attention weight of each channel, and multiplying the attention weight element-wise by the original feature map to enhance the response of the preset color.

[0019] Furthermore, the geometric constraint loss based on the spatial distribution of the weld feature points is a centroid loss function, which is calculated by measuring the difference between the centroid of the weld feature point region in the feature map generated by the generator network and the centroid of the corresponding weld feature point region in the real feature map annotation, so as to improve the geometric consistency of the generated weld feature points.

[0020] Furthermore, the calculation of the centroid loss function includes: performing connected component analysis on the weld feature point regions in the generated high-quality feature map and the real feature map annotations, identifying independent feature regions, calculating the centroid coordinates of the pixel distribution of each connected component, and using the Huber loss function to calculate the difference in centroid coordinates of the corresponding connected components.

[0021] Furthermore, the pixel-based content loss is a Smooth L1 loss function, which combines the advantages of L1 loss being insensitive to outliers and L2 loss being smooth and easy to optimize when the error is small, thus helping to improve the robustness and training stability of the model.

[0022] Furthermore, the extracted weld centerline and weld feature point information are used for real-time planning or adjustment of the robot welding path.

[0023] Furthermore, this invention also provides a multi-layer, multi-pass weld feature recognition system based on ResNet-CGAN, which is designed to implement the above method. The system includes: A laser vision sensor is configured to acquire image data including the weld centerline and weld feature points; The processor is configured to: receive image data acquired by the laser vision sensor; input the image data into a generator network, the generator network being constructed based on an encoder-decoder structure of a residual network and incorporating a color attention mechanism; the generator network being optimized and trained to generate a high-quality feature map containing enhanced weld centerlines and weld feature points; wherein, the optimized training of the generator network utilizes a discriminator network to distinguish between the high-quality feature map and real feature annotations, and performs this discrimination using a combined loss function that combines conditional generative adversarial network loss, pixel-based content loss, and geometric constraint loss based on the spatial distribution of the weld feature points; The feature extraction module is used to extract weld centerline and weld feature point information from the high-quality feature map output by the optimized generator network. The output module is used to output the weld centerline and weld feature point information for real-time planning or adjustment of the welding robot's path.

[0024] Beneficial effects: Compared with the prior art, the advantages of the present invention are: The present invention integrates the deep feature extraction capability of ResNet and the generation distribution fitting capability of CGAN, combined with SmoothL1 content loss, and can accurately recover weld details from blurred and noisy images. Experiments have shown that error indicators such as MAE and RMSE are significantly reduced.

[0025] The color attention mechanism proposed in this invention can effectively focus on the color information of key laser stripes and weld feature points, suppressing interference from strong background noise such as arc light and spatter. By introducing a centroid loss function, the generated weld feature points are directly constrained spatially, making them more consistent with the geometry of a real weld. This reduces positioning error fluctuations caused by weld bead accumulation and deformation, avoiding structural deviations that may result from relying solely on pixel-level losses. Experiments show that the performance is stable (small fluctuations in mPA and mIoU) under different weld bead layers, exhibiting strong robustness and anti-interference capabilities.

[0026] This invention achieves a high processing speed (36.72 FPS in experiments) while ensuring accuracy through a reasonable network structure design, meeting the real-time requirements of high-speed industrial welding and significantly improving the intelligence level and welding quality of multi-layer and multi-pass robotic welding. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the overall architecture of the Resnet-CGAN network provided by this invention.

[0028] Figure 2 This is a schematic diagram of the residual block structure used in this invention.

[0029] Figure 3These are error curves representing the extraction results of weld feature points in this invention, where: (a) first weld; (b) second weld; (c) third weld; (d) fourth weld; (e) fifth weld; (f) sixth weld; Figure 4 This is a schematic diagram comparing the mean absolute errors of U-net, U-net++, and Resnet-CGAN in this invention; Figure 5 This is a schematic diagram comparing the root mean square error of U-net, U-net++, and Resnet-CGAN in this invention; Figure 6 This is a schematic diagram of the visualization results of the Resnet-CGAN model training process of this invention. Detailed Implementation

[0030] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the embodiments described.

[0031] Example 1: The multi-layer, multi-pass weld feature recognition method based on ResNet-CGAN provided by this invention integrates the residual structure of Conditional Generative Adversarial Network (CGAN) and Residual Network (ResNet). ResNet-CGAN consists of a generator network G fused with a color attention mechanism C and a discriminator network D.

[0032] 1. Network Model Construction The generator network G is based on the ResNet network and incorporates an encoder-decoder pattern. A key feature of this architecture is the use of "residual blocks" to overcome the vanishing and exploding gradient problems encountered during adversarial network training. By introducing skip connections, ResNet allows information to be passed across layers, aiding in the training of deep networks and improving performance. Furthermore, a color attention mechanism is embedded in the encoder and decoder paths, dynamically focusing on important features across different color channels. The discriminator network D contains multiple convolutional layers that progressively extract spatial features from the image. Through layer-by-layer convolution, it learns features from local regions (e.g., edges, textures) to global structures (e.g., shapes, patterns), ultimately mapping these features to a scalar used to distinguish between real and fake images.

[0033] The overall architecture of the network model is as follows Figure 1 As shown, it includes an input image, a generator, and a discriminator.

[0034] The input image (Capture image) is a raw weld image captured by a laser vision sensor, with dimensions of 640*640 pixels and 3 color channels (RGB).

[0035] Generator Networks ( Figure 1 The overall structure of the main path in the upper middle section adopts an encoder-decoder architecture and integrates residual blocks and color attention (CA) mechanisms.

[0036] The generator network's task in the model is to generate high-quality feature maps from low-quality weld images, where weld feature point information can be amplified and enhanced. The generator uses a structure of downsampling layers, residual blocks, upsampling, and skip connections to ensure the ability to simultaneously represent spatial details and global features.

[0037] The input image is first downsampled, gradually halving its spatial resolution (from 1x to 1 / 8x). This process aims to extract multi-scale contextual features from the image. Color attention (CA) is applied in this path to enhance the feature response to specific colors. Residual Blocks: At the deepest layer of the network (where the spatial resolution is lowest), multiple residual blocks are used for depth feature transformation. These blocks utilize internal skip connections to help mitigate the vanishing gradient problem, enabling the network to learn more complex feature representations. Upsampling: The feature maps processed by the residual blocks are then passed through a series of upsampling modules, gradually doubling the spatial resolution (from 1 / 8x back to 1x). This process aims to reconstruct high-resolution details based on the extracted deep features and, in conjunction with any existing skip connections from the encoder to the decoder. The color attention (CA) module is also applied in this path to continue focusing on key color features during reconstruction.

[0038] The generator ultimately outputs a high-quality feature map (GenerateImage) with the same size as the input (or the target size), in which the weld centerline (red) and weld feature points (green / blue) are clearly and accurately restored and enhanced.

[0039] Discriminator Network ( Figure 1The discriminator network (including its lower-middle structure and related loss) is used during training to distinguish between "fake" images generated by the generator and real "target" images (labeled images). The discriminator network's task is to classify the input image, determining whether it is a real image or a fake image generated by the generator. The existence of the discriminator allows the generator to continuously optimize itself through adversarial training. Each time the generator generates an image, it attempts to "deceive" the discriminator into believing that the generated image is real. In this way, the generator and discriminator compete against each other, driving the model to continuously improve the quality of the generated images.

[0040] It receives a pair of images as input, extracts features from the input feature map, and finally determines its authenticity, outputting a discrimination result to calculate the discriminator loss, and then obtains the total discriminator loss (L_discriminator = l_real + l_fake).

[0041] Discriminator loss (L_discriminator): Calculated based on the discriminator's discrimination results between real and fake sample pairs, with the goal of maximizing the ability to distinguish between real and fake samples.

[0042] The generator loss (L_generator) is a combined loss used to guide the optimization of the generator; the adversarial loss (g_loss) originates from the discriminator's feedback and aims to minimize the probability of the generated image being judged as fake (i.e., "deceiving" the discriminator); the content loss (SmoothL1_loss) calculates the pixel-level difference between the generated image and the ground truth labeled image (such as the Smooth L1 distance) to ensure that the generated content is consistent with the target, and λ is its weight coefficient; the geometric loss (Centroid_loss) calculates the difference between the centroid of the weld feature point region in the generated image and the centroid of the corresponding region in the ground truth, used to constrain the spatial accuracy of the generated weld feature points, and ω is its weight coefficient.

[0043] 2. Residual Module This invention introduces "Residual Connections" into the conditional adversarial network structure, which improves its performance and stability when training deep networks. This is a simple yet highly effective design approach that allows information to bypass intermediate layers by adding skip connections between different layers of the network. Figure 2As shown, the basic unit of ResNet is the residual block. In a residual block, the input data is not only processed by ordinary convolutional layers, but also passed directly to the output of the block through a "shortcut". The output of the residual block is calculated by adding the original input and the output after the convolution operation. This means that the network can directly utilize the original input information and learn more complex features at the same time.

[0044] 3. Color Attention Mechanism In convolutional neural networks (CNNs), conventional convolution operations often treat different color channels equally. However, for many computer vision tasks, color information plays a crucial role in feature extraction. For the extraction of weld center lines and weld feature points, this invention uses red, green, and blue to label the center lines and weld feature points. Therefore, this invention designs a color attention mechanism to assign different weights to specific color channels, enabling the model to focus more on certain key color regions while suppressing unimportant parts.

[0045] The main goal of this module is to enhance the model's performance by using adaptive color weights, simulating the sensitivity of the human visual system to different colors. The module calculates global color statistics and generates color weights, marking the laser centerline and weld feature points with red, green, and blue. The color attention mechanism focuses more on the centerline and weld feature points, achieving fine-grained control over the input features.

[0046] The main design idea of ​​the color attention module is to independently generate a learnable scaling factor for each color channel (red, green, blue), thereby improving the model's color sensitivity. The weight tensor shape for each color channel is... Here, C represents the number of channels in the input features. These weight tensors serve as trainable parameters of the model and are updated during backpropagation based on the gradient of the loss function, enabling the model to gradually learn the importance of different colors. To focus more on locally salient regions, this invention employs sliding window max pooling instead of traditional global average pooling. Local max pooling extracts the maximum value within each k×k window, allowing the model to capture the highest response within that region. Compared to average pooling, max pooling pays more attention to strong responses in local regions and effectively filters noise.

[0047] (1) Represented in pixels 'b' is the batch index, and 'c' is the channel index. Through sliding window operations, the model generates a local maximum response for each pixel.

[0048] After max pooling, this invention multiplies the pooling result by the learnable weights of each channel and compresses the result to the range [0, 1] using a sigmoid activation function. After calculating the attention weights for each channel, this invention multiplies them element-wise by the original feature map, thereby enhancing the salient regions of each channel. (2) in, Let c be the c-th color channel of the input feature map. This is the Sigmoid function. This mechanism dynamically adjusts the weights of each channel to focus the model on the weld centerline (red) and weld feature points (blue / green) areas, suppressing background interference.

[0049] 4. Loss Function The choice of loss function is crucial for optimizing model training. L1 norm (absolute error) and L2 norm (mean squared error) are the most commonly used loss functions in adversarial networks. However, raw laser images of welds contain significant information interference such as reflections. While the L1 loss function effectively reduces attention to outlier locations, it also neglects weld bends, leading to larger errors. L2 loss converges quickly but focuses too much on interfering information, resulting in low accuracy in weld feature extraction and a tendency for gradient explosion. The Smooth L1 loss function combines the advantages of both L1 and L2. When the difference between the predicted and actual values ​​is small (less than a certain threshold δ), Smooth L1 behaves like L1; however, when the difference is large (greater than the threshold δ), it transforms into L2, reducing sensitivity to outliers. This design allows Huber Loss to maintain the efficiency of L2 in most cases while retaining the robustness of L1 when outliers occur.

[0050] (3) In the identification of weld feature points in multi-layer, multi-pass welds, to improve the accuracy of weld feature point location in images generated by Conditional Generative Adversarial Networks (CGAN), this paper designs a centroid loss function as an auxiliary loss term and incorporates it into the generator training. This loss function, by calculating the difference in centroid positions between the generated and real images of weld feature point regions, guides the generator to more accurately generate weld feature points that conform to the actual weld distribution, directly constraining the spatial distribution of weld feature points in the generated and real images, making the generated weld feature points more consistent with the actual weld shape and location. After the weld feature points are stably generated, the weld feature point regions in the green and blue channels are extracted from the generated and real images, respectively. Connected component labeling is performed on the extracted weld feature point regions to identify independent weld feature regions. For each connected component, the centroid coordinates of its pixel distribution are calculated, and then the Huber loss function is used to calculate the difference in centroid coordinates between the corresponding connected components in the generated and real images, as shown in the following formula: (4) in, and Let G be the centroid coordinates of the generated and ground images of the i-th connected component, respectively. Through connected component analysis and centroid calculation, this method can effectively identify the complex distribution of weld feature points in multi-layer, multi-pass weld images, avoiding feature loss due to noise or excessive smoothing. The network combines GAN loss, L1 loss, and centroid loss to optimize G: (5) in, and This is the loss weight.

[0051] Example 2: To improve network reliability, this example utilizes a Yaskawa robot 3D scanning device to collect a real dataset. This invention collects six types of weld samples, ranging from 1 layer with 1 pass to 3 layers with 6 passes, for experiments, collecting a total of 900 images with a resolution of 1280*960 pixels as the training dataset. To avoid introducing excessive noise and useless information that could affect training efficiency, a region-of-interest (ROI) model is used to extract images with a resolution of 640*640 pixels as the network input.

[0052] The model was developed using Python 3.11, Torch 2.0.0, and the cu118 library. Windows 11 was used as the development platform, equipped with an Intel(R) Core(TM) i7-14700KF CPU and an NVIDIA GeForce RTX-4090 GPU for training and testing. The neural network had an initial learning rate of 0.0005, which was adaptively optimized using the Adam optimizer.

[0053] To verify the effectiveness of laser line and weld feature point extraction, the mean pixel accuracy (mPA) and mean intersection pair union (mIOU) of each weld bead are quantitatively evaluated based on the network proposed in this invention. Unlike ordinary pixel evaluation, since the image contains a large amount of black background, mPA and mIOU evaluation are meaningless. Therefore, this invention uses red, green, and blue as index categories to evaluate the weld centerline, weld feature points at the bottom of the weld bead, and weld feature points at the top of the weld bead in the generated image.

[0054] (6) (7) Meanwhile, in order to verify the accuracy of the weld centerline and weld feature points, the mean absolute error (MAE) of the laser lines and weld feature points in 30 weld centerline images was evaluated.

[0055] (8) To further verify the advantages of the proposed network, this invention was compared with existing network architectures such as U-net and U-net++. The prediction results of the three networks for weld feature information were quantitatively evaluated using root mean square error (RMSE) and mean absolute error (MAE).

[0056] Evaluation results: As shown in Table 1, as the number of weld layers increases (Layer 1 to Layer 6), mPA and mIoU only fluctuate by ±3%, indicating that the model has strong robustness to multi-layer stacking scenarios. The decrease in accuracy of Layer 4 (mPA=89.8%) is due to the blurring of local features caused by weld slag accumulation in this layer, but the model's anti-interference ability is verified by the centroid loss function constraint.

[0057] Table 1. Number of Weld Layers and Corresponding mPA and mIoU Evaluation Table Weld pass mPA / % mIoU / % Layer1 93.7 87.4 Layer2 94.0 87.6 Layer 3 94.4 88.4 Layer 4 89.8 83.1 Layer 5 92.8 85.9 Layer 6 90.8 83.9 like Figure 3 As shown, the ResNet network has a good ability to extract feature points of each weld layer. For images with gentle slope angles, the mean absolute error is relatively high, generally remaining within 2 pixels. The average error of each layer does not exceed 0.6 pixels.

[0058] like Figure 4 , Figure 5As shown, the MAE and RMSE of ResNet-CGAN (corresponding to ours in the figure) provided by this invention are significantly lower than those of the U-net series (U-net++, U-net). This is because: 1) Residual connections alleviate gradient vanishing in deep networks, preserving more detailed features; 2) Adversarial training of CGAN enhances the geometric consistency between the generated image and the real distribution; 3) The color attention mechanism effectively suppresses arc light interference (such as...) through dynamic weight allocation. Figure 5 (As shown). In addition, U-net++ introduces redundant parameters due to dense skip connections, resulting in insufficient real-time performance (FPS=28.5), while ResNet-CGAN balances accuracy and efficiency through a lightweight design.

[0059] Among the three network models, the ResNet-CGAN network exhibits the lowest MAE and RMSE errors and the smallest curve fluctuations, indicating that its feature extraction capability is the best among the three, demonstrating strong robustness. Visualization results are shown below. Figure 6 As shown.

[0060] Example 3: This invention proposes a system for implementing a multi-layer, multi-pass weld feature recognition method based on ResNet-CGAN. The system includes: Laser vision sensor: Installed on the robot's end effector to acquire images of the weld seam.

[0061] Processor: An application program for implementing the method in Example 1 (including a pre-trained generator G model, a discriminator D model, a loss function calculation module, and a feature extraction module).

[0062] Robot controller: Receives weld feature information output by the processing unit and controls the movement of the welding robot.

[0063] Workflow: The laser vision sensor acquires weld seam images and transmits them to the processor. The processor executes the application program, calls the pre-trained generator G model to process the images, and generates high-quality feature maps. The feature extraction module extracts laser stripe and feature point information from the feature maps. The processor sends the information to the robot controller, which adjusts the posture and trajectory of the welding robot's welding torch accordingly to perform precise welding.

[0064] This invention has the following advantages: by fusing deep features with ResNet-CGAN and adversarial training, the MAE is reduced by 30%, achieving high-precision extraction of weld information; the color attention mechanism suppresses arc light and spatter noise, and the weld feature point positioning error fluctuation is <0.6 pixels, which has strong anti-interference ability; the lightweight design enables a frame rate of 36.72 FPS, supporting high-speed welding scenarios and excellent real-time performance; the centroid loss function ensures that the generated weld feature points match the spatial distribution of the real weld, with geometric consistency constraints.

[0065] As described above, although the invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the invention itself. Various changes in form and detail may be made without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for feature recognition of multi-layer, multi-pass weld seams based on ResNet-CGAN, characterized in that, Includes the following steps: Acquire image data of multi-layer, multi-pass welding; The image data is input into a pre-trained model that fuses a residual network and a conditional generative adversarial network, the model including a generator network and a discriminator network; The generator network is constructed based on the encoder-decoder structure of a residual network and incorporates a color attention mechanism. The color attention mechanism uses the color channels used to represent the weld centerline, the weld feature points at the bottom of the weld bead, and the weld feature points at the top of the weld bead as preset color channels. A learnable scaling factor is set independently for each preset color channel. Sliding window local max pooling is performed on each preset color channel to extract the highest response in the local region. The output of the sliding window local max pooling is multiplied by the learnable scaling factor of the corresponding preset color channel, and the attention weights of each preset color channel are obtained by processing with the Sigmoid activation function. The attention weights are multiplied element-wise by the corresponding original feature map in the generator network to enhance the regions where the weld centerline, the weld feature points at the bottom of the weld bead, and the weld feature points at the top of the weld bead are located, generating a high-quality feature map containing the enhanced weld centerline and weld feature points. The discriminator network discriminates between the high-quality feature map and the ground truth labeled feature map generated by the generator network. It then optimizes the generator network by combining a loss function that integrates conditional generative adversarial network loss, pixel-based content loss, and geometric constraint loss based on the spatial distribution of weld feature points. The geometric constraint loss is a centroid loss. This centroid loss is achieved by performing connected component analysis on the corresponding weld feature point regions in the high-quality feature map and the ground truth labeled feature map, respectively, for weld feature points at the bottom and top of the weld bead. This process identifies independent feature regions and calculates the centroid coordinates of the pixel distribution within these regions. Finally, the Huber loss function is used to calculate the difference in centroid coordinates between the high-quality feature map generated by the generator network and the ground truth labeled feature map for the same category of weld feature point regions. Weld centerline and weld feature point information are extracted from the high-quality feature map output by the optimized generator network.

2. The method for multi-layer, multi-pass weld feature recognition based on ResNet-CGAN according to claim 1, characterized in that, The generator network includes an encoder, residual blocks, and a decoder. The encoder and the decoder pass shallow detail features through skip connections, and the residual blocks pass input features through residual connections. Both the encoder and the decoder are embedded with the color attention mechanism.

3. The method for multi-layer, multi-pass weld feature recognition based on ResNet-CGAN according to claim 1, characterized in that, The preset color channel used to characterize the center line of the weld is the red channel, the preset color channel used to characterize the weld feature points at the bottom of the weld bead is the green channel, and the preset color channel used to characterize the weld feature points at the top of the weld bead is the blue channel.

4. The method for multi-layer, multi-pass weld feature recognition based on ResNet-CGAN according to claim 1, characterized in that, The pixel-based content loss is the Smooth L1 loss function.

5. The method for multi-layer, multi-pass weld feature recognition based on ResNet-CGAN according to claim 1, characterized in that, The extracted weld centerline and weld feature point information are used for real-time planning or adjustment of the welding path of the welding robot.

6. A multi-layer, multi-pass weld feature recognition system based on ResNet-CGAN, characterized in that, The system includes a laser vision sensor, a processor, a feature extraction module, and an output module. The laser vision sensor is configured to acquire multi-layer, multi-pass welding image data to be identified. The processor is configured to receive the multi-layer, multi-pass welding image data to be identified and input it into an optimized and trained generator network. The generator network is constructed based on an encoder-decoder structure of a residual network and incorporates a color attention mechanism. The color attention mechanism uses color channels representing the weld centerline, weld feature points at the bottom of the weld bead, and weld feature points at the top of the weld bead as preset color channels. A learnable scaling factor is independently set for each preset color channel. Sliding window local max pooling is performed on each preset color channel to extract the highest response in the local region. The output of the sliding window local max pooling is multiplied by the learnable scaling factor of the corresponding preset color channel, and the attention weights of each preset color channel are obtained through a Sigmoid activation function. The intention weights are multiplied element-wise by the corresponding original feature map in the generator network to generate a feature map containing the enhanced weld centerline, the weld feature points at the bottom of the weld bead, and the weld feature points at the top of the weld bead. The generator network is a network obtained by using a discriminator network to distinguish between the feature map generated by the generator network and the real labeled feature map, and by optimizing and training a combined loss function that combines conditional generative adversarial network loss, pixel-based content loss, and centroid loss. The centroid loss is achieved by performing connected component analysis on the corresponding weld feature point regions in the feature map generated by the generator network and the real labeled feature map for the weld feature points at the bottom and top of the weld bead, respectively, to identify independent feature regions and calculate the centroid coordinates of the pixel distribution in the feature regions, and by using the Huber loss function to calculate the difference in centroid coordinates between the feature map generated by the generator network and the weld feature point regions of the same category in the real labeled feature map. The feature extraction module is used to extract the weld centerline and weld feature point information from the feature map output by the generator network; the output module is used to output the weld centerline and weld feature point information for the welding robot to perform real-time planning or adjustment of the welding path.

Images