TIG (Tungsten Inert Gas) welding pool image segmentation method based on Mamba and YOLO11

Abstract

The invention discloses a TIG welding pool image segmentation method based on Mamba and YOLO11, and relates to the technical field of pool image processing. The invention provides a real-time and efficient segmentation method, a deep learning model can be optimized and accelerated by using ONNX and TensorRT model deployment frameworks, and light weight and efficient reasoning of the model are realized. Therefore, real-time segmentation can be realized on resource-limited equipment, the segmentation precision and reliability can be ensured, and the method is favorable for acquiring key characteristic parameters in real time and finding potential quality problems in time in the welding process, so that the online monitoring and control of the welding quality are realized, the product quality is improved, the production cost is reduced, and the production efficiency is improved. And technical support is provided for intelligent manufacturing and industrial automation of a steam turbine plant.

Description

Technical Field

[0001] The present invention relates to the technical field of molten pool image processing, and in particular to a TIG welding molten pool image segmentation method based on Mamba and YOLO11. Background Art

[0002] Tungsten inert gas welding (TIG welding) is widely used in high-precision manufacturing fields, such as the production process of steam turbines. Accurate segmentation of welding pool images is essential for online monitoring and quality control of the welding process. However, welding pool images have the characteristics of data diversity, complex background and many interference factors, such as arc light, welding smoke, and ambient light changes. These factors lead to the degradation of image quality, noise, blur and concealment, which increases the difficulty of image segmentation.

[0003] With the rapid development of deep learning, image segmentation methods based on convolutional neural networks (CNN), such as U-Net, SegNet, and DeepLab, have been widely used in medical imaging, remote sensing images, and other fields. These methods can extract rich features and achieve high-precision segmentation through deep network structures and a large number of parameters. However, these models usually have a large number of parameters and high computational complexity, require high-performance hardware support, and are difficult to implement in real time in resource-constrained industrial sites. At the same time, these models are mainly trained on natural images or data in specific fields, and direct application to welding pool images may not achieve ideal results.

[0004] As one of the mainstream methods for target detection, the YOLO algorithm has attracted widespread attention due to its high computing speed, low computational complexity, and high accuracy. In natural image target detection tasks, YOLO performs well. However, when the YOLO algorithm is directly applied to welding pool images, the effect is not ideal. The main reason is that the complexity and particularity of welding pool images make the YOLO algorithm weak in learning the features of the molten pool area and difficult to adapt to the diversity of data and illumination changes. As a selective state space model, the Mamba model has attracted much attention due to its ability to model long-distance dependencies under linear complexity. Mamba was originally designed to handle sequence-to-sequence learning tasks, but its application potential in the field of computer vision has gradually been discovered. Combining Mamba with traditional target detection models is expected to improve the accuracy and efficiency of welding pool image segmentation. By integrating Mamba's long-distance dependency modeling capabilities and YOLO's efficient detection characteristics, the model's ability to learn the features of the molten pool area can be enhanced while maintaining a high inference speed.

[0005] However, there is no mature solution in this field and related research is relatively limited. This provides us with new research directions and innovation space to explore how to effectively integrate the Mamba model with the YOLO algorithm and optimize the design based on the characteristics of the welding pool image.

[0006] In summary, the existing technology has the following deficiencies in welding pool image segmentation:

[0007] 1. Poor adaptability to complex environments: During the welding process, the molten pool image is easily affected by various interferences such as arc light, welding smoke, and ambient light, resulting in reduced image quality, noise, and blur. This makes it difficult to accurately segment and extract features of the molten pool area, which in turn affects the accuracy of welding defect detection. Traditional methods and directly applied deep learning models are difficult to effectively deal with complex interferences such as arc light, smoke, and changes in lighting during welding, resulting in reduced segmentation accuracy.

[0008] 2. High model complexity, difficult to apply in real time: In actual industrial applications, computing resources and hardware environments are often limited. Due to the large number of parameters and high computational complexity, deep learning models are difficult to implement real-time reasoning on resource-constrained devices. This not only increases the deployment cost, but also limits the scope of application of the model. Many high-precision deep learning models have large parameters and high computational complexity, and cannot be implemented in real-time reasoning on resource-constrained devices at industrial sites, limiting their scope of application.

[0009] 3. Insufficient feature extraction capabilities: The welding process in a steam turbine plant is complex and the environment is changeable. The welding quality is directly related to the performance and safety of the product. Traditional welding quality detection methods are often lagging and unable to detect and correct welding defects in a timely manner. When processing welding pool images, existing models have limited ability to capture multi-scale features and contextual information, and cannot fully extract the key features of the molten pool area.

[0010] 4. Lack of model design for welding molten pool: Most models are not optimized for the particularity of welding molten pool, and lack in-depth exploration and utilization of the characteristics of the molten pool area.

[0011] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0012] The purpose of the present invention is to provide a TIG welding molten pool image segmentation method based on Mamba and YOLO11 to solve the technical problems raised in the background technology.

[0013] To achieve the above object, the present invention provides the following technical solution: a TIG welding molten pool image segmentation method based on Mamba and YOLO11, comprising at least the following steps:

[0014] Step 1: Use the developed acquisition software to obtain the TIG welding pool image data of the turbine plant and divide it into training set, validation set and test set;

[0015] Step 2: Perform data augmentation and preprocessing on the dataset;

[0016] Step 3: Design the neural network models of Mamba and YOLO11, put the processed data set into the designed neural network model for training and verification, and save the neural network model;

[0017] Step 4: Convert the trained model to ONNX and TensorRT for deployment.

[0018] Furthermore, the step 1 at least includes the following steps:

[0019] Step 1.1: Use homemade data acquisition equipment and develop software for automatic data acquisition;

[0020] Using homemade acquisition equipment and developed software, we automatically acquired image data of the TIG welding pool in the steam turbine plant;

[0021] Ensure that the captured images have high resolution and clarity through equipment adjustment and software optimization;

[0022] Step 1.2: Use the CVAT platform to store the dataset and automatically label it;

[0023] Use the CVAT (Computer Vision Annotation Tool) platform to manage the data set, and the CVAT platform is built on its own server;

[0024] By importing the Seg2 model, the welding pool area is marked using the automatic marking tool;

[0025] Step 1.3: Convert format;

[0026] Convert the annotation files exported by CVAT to YOLO format to adapt to the deep learning framework (PyTorch) and ensure data format compatibility to maintain consistency during training and inference;

[0027] Step 1.4: Divide the data set into a training set, a validation set, and a test set, wherein the training set, the validation set, and the test set are divided according to a ratio of 9:0.5:0.5.

[0028] Furthermore, the step 2 at least includes the following steps:

[0029] Data augmentation, using data augmentation techniques to improve the robustness of the model, the data augmentation techniques at least include random cropping, rotation and scaling;

[0030] Data preprocessing: uniformly adjust the image size and perform normalization; apply the CLAHE (contrast limited adaptive histogram equalization) method to enhance the image contrast to better highlight the melt pool features; convert to tensor format, and adjust the data channel order to adapt to the model input.

[0031] 10. The TIG welding molten pool image segmentation method based on Mamba and YOLO11 according to claim 1, characterized in that: the neural network model of Mamba and YOLO11 in step 3 at least comprises the following steps:

[0032] Step 3.1: Design an enhanced state space block, which is an ESSB module. The purpose of designing the ESSB module is to enhance the model's ability to capture multi-scale information and complex contexts through multi-level feature extraction and fusion;

[0033] Step 3.2: Use the Simple Stem module and CBS module to quickly downsample the input image in the initial stage and efficiently extract the initial features to ensure that the input image is effectively preprocessed before entering the deep network;

[0034] Step 3.3: In the YOLO11 network, the original C3k2 module is replaced with the ESSB module. With the introduction of the ESSB module, the model can better capture multi-scale features and complex contextual information, thereby improving the detection performance. At the same time, due to the extensive use of deep separable convolutions and modular design in the ESSB module, the inference speed of the model is significantly improved.

[0035] Furthermore, the ESSB (Enhanced State-Space Block) module consists of three core sub-modules: LFA (Local Feature Aggregation), RGU (Residual Gated Unit) and SS2D (Selective-Scan 2DBlock).

[0036] Furthermore, the application of the ESSB module comprises at least the following steps:

[0037] After the initial convolution, batch normalization and activation layer, it goes to the designed LFA module; after layer normalization, the features are standardized to ensure the consistency of data distribution. The feature map after layer normalization enters the SS2D module, and then a residual connection is performed to weight the SS2D output and the features of the previous layer. After layer normalization again, it goes to the designed RGU module. The output of the RGU module is again fused with the features of the previous layer through an addition operator to form the final output;

[0038] The application process of the LFA module is as follows: the input features are firstly extracted through a C3k2 module, then the features are normalized through batch normalization, and then the nonlinear expression ability is increased through a convolution layer and an activation layer, and finally the input features are added to the output features through a convolution layer and a residual connection to retain the original information while enhancing the feature expression ability;

[0039] The application process of SS2D is as follows: the input first passes through a linear layer, then is further processed by a depthwise separable convolution (DW-Conv) and an activation layer, then features are extracted by a Scan operation, and finally passes through a linear layer and layer normalization to output the final features;

[0040] The application process of the RGU is as follows: the input features first pass through a convolution layer to extract preliminary features, and then enter the gating mechanism. In the gating mechanism, the features are divided into two paths, one of which directly transmits the features, and the other path processes the features through depthwise separable convolution (DW-Conv) and activation functions. The features processed by DW-Conv and the features of the direct path are multiplied in the gating unit to form a control signal. Then, the control signal is added to the initial input features and the final result is output. Through the gating mechanism, the RGU module can selectively control feature transmission, so that the network can more effectively extract key information at different levels.

[0041] Furthermore, the algorithmic representation of the ESSB module includes at least the following steps:

[0042] Initial convolution, batch normalization, and activation:

[0043] Input feature F in First, it passes through a convolution layer, batch normalization, and activation function (SiLU):

[0044]

[0045] Among them: BN is batch normalization, is the activation function, the SiLU activation function is used here;

[0046] Design of LFA module:

[0047] Use C3k2 module for preliminary feature extraction:

[0048] F1=C3k2(F0)

[0049] Batch Normalization:

[0050] F2=BN(F1)

[0051] Non-linear mapping:

[0052]

[0053] Residual Connection:

[0054] F LFA =F0+Conv(F3)

[0055] The output of the LFA module is subjected to layer normalization, which is expressed as LN:

[0056] F LN1 =LN(F LFA )

[0057] SS2D module:

[0058] F SS2D =SS2D(F LN1 )+F LN1

[0059] Residual connections and weighted fusion:

[0060] The output of SS2D is weighted and fused with the features of the previous layer:

[0061] F merge1 =F LN1 +α·F SS2D

[0062] Among them, α is a learnable weight parameter;

[0063] Normalize the layer again:

[0064] F LN2 =LN(F merge1 )

[0065] RGU Modules:

[0066] Initial convolution:

[0067] X1=Conv(F LN2 )

[0068] Feature branches:

[0069] Branch 1 (direct transfer):

[0070] X α =X1

[0071] Branch 2 (depthwise separable convolution and activation):

[0072] The input feature X1 is first mixed by 1×1 convolution, then the spatial features are extracted by 3×3 depth-separable convolution, and finally the nonlinear transformation is performed by SiLU activation function, and the output X β :

[0073]

[0074] Where: DW_Conv 3·3 It is a 3×3 depth-wise separable convolution;

[0075] Gating mechanism:

[0076] Multiply the results of the two branches element-wise:

[0077] X gated =X α ⊙X β

[0078] Fusion output and residual connection:

[0079] F RGU =F LN2 +Conv 1·1 (X gated )

[0080] Final output:

[0081] X out =F RGU .

[0082] Furthermore, the step 3.2 at least includes the following steps:

[0083] The Simple Stem module is used to quickly downsample the input image in the initial stage and efficiently extract the initial features;

[0084] The Simple Stem module includes a first layer of convolution and batch normalization and a second layer of convolution and batch normalization;

[0085] The first layer of convolution and batch normalization is expressed as follows:

[0086]

[0087] The second layer of convolution and batch normalization is expressed as follows:

[0088]

[0089] The CBS module is used to further enhance feature extraction and fusion in the Neck part, as shown below:

[0090]

[0091] N in It is the feature map output after being processed by the ESSB module.

[0092] Furthermore, the step 4 at least includes the following steps:

[0093] Step 4.1: Convert the model to ONNX format: Export the trained model in PyTorch format to ONNX format to ensure the compatibility of the model between different frameworks;

[0094] Step 4.2: Use TensorRT to optimize the model. Convert the ONNX model to the TensorRT engine, perform deep optimization for NVIDIA GPU, and use INT8 quantization technology to further improve the model's reasoning performance, reduce latency, and meet the needs of real-time applications.

[0095] Step 4.3: Deploy the model to the production environment for comparison.

[0096] Compared with the prior art, the present invention has the following beneficial effects:

[0097] 1. The present invention provides a real-time and efficient segmentation method. By using ONNX and TensorRT model deployment framework, the deep learning model can be optimized and accelerated to achieve lightweight and efficient reasoning of the model. In this way, not only can real-time segmentation be achieved on resource-constrained devices, but also the accuracy and reliability of segmentation can be ensured. This method helps to obtain key feature parameters in real time during the welding process and promptly discover potential quality problems, thereby achieving online monitoring and control of welding quality, improving product quality, reducing production costs, and providing technical support for intelligent manufacturing and industrial automation in steam turbine plants;

[0098] 2. The present invention can significantly improve the reasoning speed by reducing the number of model parameters and optimizing the network structure, and realize the rapid deployment and application of the model in the industrial field under the premise of ensuring the segmentation accuracy. This is of great significance for meeting the real-time requirements of the industrial field and improving production efficiency, thereby achieving the effect of lightweight model, reducing parameters and improving reasoning speed;

[0099] 3. The present invention solves the problem of how to achieve high-precision feature extraction of molten pool images in complex industrial environments, and improves the accuracy of molten pool image segmentation through method design, especially under complex lighting and backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0101] Figure 1 This is the Mamba-YOLO11 network architecture diagram of the present invention;

[0102] Figure 2 It is a composite schematic diagram of the present invention, wherein Figure 2 (a) is a schematic diagram of the ESSB framework; Figure 2 (b) is a schematic diagram of the local feature aggregation block LFAB; Figure 2 (c) is a schematic diagram of the residual gate control unit RGU; Figure 2 (d) is a schematic diagram of the SS2D structure; Figure 2 (e) is a schematic diagram of the C3k2 module in YOLO11; Figure 2 (f) is a schematic diagram of the C3 module;

[0103] Figure 3 The structure diagram of the Simple stem module and the CBS module of the present invention is shown in FIG.

[0104] Figure 4 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0105] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0106] See also Figure 1-Figure 3 , the TIG welding molten pool image segmentation method based on Mamba and YOLO11 at least includes the following steps:

[0107] Step 1: Use the developed acquisition software to obtain the TIG welding pool image data of the turbine plant and divide it into training set, validation set and test set;

[0108] Step 2: Perform data augmentation and preprocessing on the dataset;

[0109] Step 3: Design the neural network models of Mamba and YOLO11, put the processed data set into the designed neural network model for training and verification, and save the neural network model;

[0110] Step 4: Convert the trained model to ONNX and TensorRT for deployment.

[0111] Step 1 includes at least the following steps:

[0112] Step 1.1: Use homemade data acquisition equipment and develop software for automatic data acquisition;

[0113] Using homemade acquisition equipment and developed software, we automatically acquired image data of the TIG welding pool in the steam turbine plant;

[0114] Ensure that the captured images have high resolution and clarity through equipment adjustment and software optimization;

[0115] Step 1.2: Use the CVAT platform to store the dataset and automatically label it;

[0116] Use the CVAT (Computer Vision Annotation Tool) platform to manage the data set, and the CVAT platform is built on its own server;

[0117] By importing the Seg2 model, the welding pool area is marked using the automatic marking tool;

[0118] Step 1.3: Convert format;

[0119] Convert the annotation files exported by CVAT to YOLO format to adapt to the deep learning framework (PyTorch) and ensure data format compatibility to maintain consistency during training and inference;

[0120] Step 1.4: Divide the dataset into training set, validation set and test set. The training set, validation set and test set are divided according to 9:0.5:0.5.

[0121] Step 2 includes at least the following steps:

[0122] Data augmentation, using data augmentation techniques to improve the robustness of the model. Data augmentation techniques include at least random cropping, rotation, and scaling;

[0123] Data preprocessing: uniformly adjust the image size and perform normalization; apply the CLAHE (contrast limited adaptive histogram equalization) method to enhance the image contrast to better highlight the melt pool features; convert to tensor format, and adjust the data channel order to adapt to the model input.

[0124] 11. The TIG welding molten pool image segmentation method based on Mamba and YOLO11 according to claim 1, characterized in that: the neural network model of Mamba and YOLO11 in step 3 at least comprises the following steps:

[0125] Step 3.1: Design an enhanced state space block, namely the ESSB module. The purpose of the ESSB module is to enhance the model’s ability to capture multi-scale information and complex contexts through multi-level feature extraction and fusion;

[0126] Step 3.2: Use the Simple Stem module and CBS module to quickly downsample the input image in the initial stage and efficiently extract the initial features to ensure that the input image is effectively preprocessed before entering the deep network;

[0127] Step 3.3: In the YOLO11 network, the original C3k2 module is replaced with the ESSB module. With the introduction of the ESSB module, the model can better capture multi-scale features and complex contextual information, thereby improving the detection performance. At the same time, due to the extensive use of deep separable convolutions and modular design in the ESSB module, the inference speed of the model is significantly improved.

[0128] The ESSB (Enhanced State-Space Block) module consists of three core sub-modules: LFA (Local Feature Aggregation), RGU (Residual Gated Unit) and SS2D (Selective-Scan 2D Block).

[0129] Application of the ESSB module includes at least the following steps:

[0130] After the initial convolution, batch normalization and activation layer are passed to the designed LFA module; after layer normalization to standardize the features to ensure the consistency of data distribution, the feature map after layer normalization enters the SS2D module, and then a residual connection is performed to weight the SS2D output and the previous layer feature. After layer normalization again, it enters the designed RGU module. The output of the RGU module is again fused with the features of the previous layer through an addition operator to form the final output;

[0131] The application process of the LFA module is as follows: the input features first pass through a C3k2 module for preliminary feature extraction, then normalize the features through batch normalization, then pass through a convolution layer and an activation layer to increase the nonlinear expression ability, and finally add the input features to the output features through a convolution layer and a residual connection to retain the original information while enhancing the feature expression ability;

[0132] The application process of SS2D is as follows: the input first passes through a linear layer, then is further processed by a depth-wise separable convolution (DW-Conv) and an activation layer, then features are extracted by a Scan operation, and finally, it passes through a linear layer and layer normalization to output the final features;

[0133] The application process of RGU is as follows: the input features first pass through a convolution layer to extract preliminary features, and then enter the gating mechanism. In the gating mechanism, the features are divided into two paths, one of which directly transmits the features, and the other path processes the features through depth-separable convolution (DW-Conv) and activation function. The features processed by DW-Conv and the features of the direct path are multiplied in the gating unit to form a control signal. Then, the control signal is added to the initial input features and the final result is output. Through the gating mechanism, the RGU module can selectively control feature transmission, so that the network can more effectively extract key information at different levels.

[0134] The algorithmic representation of the ESSB module includes at least the following steps:

[0135] Initial convolution, batch normalization, and activation:

[0136] Input feature F in First, it passes through a convolution layer, batch normalization, and activation function (SiLU):

[0137]

[0138] Among them: BN is batch normalization, is the activation function, the SiLU activation function is used here;

[0139] Design of LFA module:

[0140] Use C3k2 module for preliminary feature extraction:

[0141] F1=C3k2(F0)

[0142] Batch Normalization:

[0143] F2=BN(F1)

[0144] Non-linear mapping:

[0145]

[0146] Residual Connection:

[0147] F LFA =F0+Conv(F3)

[0148] The output of the LFA module is layer normalized, and the layer normalization is expressed as LN:

[0149] F LN1 =LN(F LFA )

[0150] SS2D module:

[0151] F SS2D =SS2D(FLN1 )+F LN1

[0152] Residual connections and weighted fusion:

[0153] The output of SS2D is weighted and fused with the features of the previous layer:

[0154] F merge1 =F LN1 +α·F SS2D

[0155] Among them, α is a learnable weight parameter;

[0156] Normalize the layer again:

[0157] F LN2 =LN(F merge1 )

[0158] RGU Modules:

[0159] Initial convolution:

[0160] X1=Conv(F LN2 )

[0161] Feature branches:

[0162] Branch 1 (direct transfer):

[0163] X α =X1

[0164] Branch 2 (depthwise separable convolution and activation):

[0165] The input feature X1 is first mixed by 1×1 convolution, then the spatial features are extracted by 3×3 depth-separable convolution, and finally the nonlinear transformation is performed by SiLU activation function, and the output X β :

[0166]

[0167] Where: DW_Conv 3·3 It is a 3×3 depth-wise separable convolution;

[0168] Gating mechanism:

[0169] Multiply the results of the two branches element-wise:

[0170] X gated =X α ⊙X β

[0171] Fusion output and residual connection:

[0172] F RGU =FLN2 +Conv 1·1 (X gated )

[0173] Final output:

[0174] X out =F RGU .

[0175] Step 3.2 includes at least the following steps:

[0176] The Simple Stem module is used to quickly downsample the input image in the initial stage and efficiently extract the initial features;

[0177] The Simple Stem module includes the first layer of convolution and batch normalization and the second layer of convolution and batch normalization;

[0178] The first layer of convolution and batch normalization is expressed as follows:

[0179]

[0180] The second layer of convolution and batch normalization is expressed as follows:

[0181]

[0182] The CBS module is used to further enhance feature extraction and fusion in the Neck part, as shown below:

[0183]

[0184] N in It is the feature map output after being processed by the ESSB module.

[0185] Step 4 includes at least the following steps:

[0186] Step 4.1: Convert the model to ONNX format: Export the trained model in PyTorch format to ONNX format to ensure the compatibility of the model between different frameworks;

[0187] Step 4.2: Use TensorRT to optimize the model. Convert the ONNX model to the TensorRT engine, perform deep optimization for NVIDIA GPU, and use INT8 quantization technology to further improve the model's reasoning performance, reduce latency, and meet the needs of real-time applications.

[0188] Step 4.3: Deploy the model to the production environment for comparison.

[0189] Refer to Table 1 for the comparison between the research model and YOLOv8 and YOLO11;

[0190] Table 1 Comparison table of research models with YOLOv8 and YOLO11

[0191]

[0192] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A TIG welding molten pool image segmentation method based on Mamba and YOLO11, characterized by: At least the following steps are included: Step 1: Use the developed acquisition software to obtain the TIG welding pool image data of the turbine plant and divide it into training set, validation set and test set; Step 2: Perform data augmentation and preprocessing on the dataset; Step 3: Design the neural network models of Mamba and YOLO11, put the processed data set into the designed neural network model for training and verification, and save the neural network model; Step 4: Convert the trained model to ONNX and TensorRT for deployment.

2. The TIG welding molten pool image segmentation method based on Mamba and YOLO11 according to claim 1 is characterized in that: The step 1 at least comprises the following steps: Step 1.1: Use homemade data acquisition equipment and develop software for automatic data acquisition; Using homemade acquisition equipment and developed software, we automatically acquired image data of the TIG welding pool in the steam turbine plant; Ensure that the captured images have high resolution and clarity through equipment adjustment and software optimization; Step 1.2: Use the CVAT platform to store the dataset and automatically label it; Use the CVAT platform to manage the data set, and the CVAT platform is built on its own server; By importing the Seg2 model, the welding pool area is marked using the automatic marking tool; Step 1.3: Convert format; Convert the annotation files exported by CVAT to YOLO format to adapt to the deep learning framework (PyTorch) and ensure data format compatibility to maintain consistency during training and inference; Step 1.4: Divide the data set into a training set, a validation set, and a test set, wherein the training set, the validation set, and the test set are divided according to a ratio of 9:0.5:0.

5.

3. The TIG welding molten pool image segmentation method based on Mamba and YOLO11 according to claim 1 is characterized in that: The step 2 at least comprises the following steps: Data augmentation, using data augmentation techniques to improve the robustness of the model, the data augmentation techniques at least include random cropping, rotation and scaling; Data preprocessing: uniformly adjust the image size and perform normalization; apply the CLAHE method to enhance the image contrast to better highlight the melt pool features; convert to tensor format, and adjust the data channel order to adapt to the model input.

4. The TIG welding molten pool image segmentation method based on Mamba and YOLO11 according to claim 1 is characterized in that: Designing the neural network model of Mamba and YOLO11 in step 3 at least includes the following steps: Step 3.1: Design an enhanced state space block, which is an ESSB module. The purpose of designing the ESSB module is to enhance the model's ability to capture multi-scale information and complex contexts through multi-level feature extraction and fusion; Step 3.2: Use the Simple Stem module and CBS module to quickly downsample the input image in the initial stage and efficiently extract the initial features to ensure that the input image is effectively preprocessed before entering the deep network; Step 3.3: In the YOLO11 network, the original C3k2 module is replaced with the ESSB module. With the introduction of the ESSB module, the model can better capture multi-scale features and complex contextual information, thereby improving the detection performance. At the same time, due to the extensive use of deep separable convolutions and modular design in the ESSB module, the inference speed of the model is significantly improved.

5. The TIG welding molten pool image segmentation method based on Mamba and YOLO11 according to claim 4 is characterized in that: The ESSB module consists of three core submodules: LFA, RGU and SS2D.

6. The TIG welding molten pool image segmentation method based on Mamba and YOLO11 according to claim 5 is characterized in that: The application of the ESSB module comprises at least the following steps: After the initial convolution, batch normalization and activation layer, it goes to the designed LFA module; after layer normalization, the features are standardized to ensure the consistency of data distribution. The feature map after layer normalization enters the SS2D module, and then a residual connection is performed to weight the SS2D output and the features of the previous layer. After layer normalization again, it goes to the designed RGU module. The output of the RGU module is again fused with the features of the previous layer through an addition operator to form the final output; The application process of the LFA module is as follows: the input features are firstly extracted through a C3k2 module, then the features are normalized through batch normalization, and then the nonlinear expression ability is increased through a convolution layer and an activation layer, and finally the input features are added to the output features through a convolution layer and a residual connection to retain the original information while enhancing the feature expression ability; The application process of SS2D is as follows: the input first passes through a linear layer, then is further processed through a depth-wise separable convolution and activation layer, then features are extracted through a Scan operation, and finally passes through a linear layer and layer normalization to output the final features; The application process of the RGU is as follows: the input features first extract preliminary features through a convolution layer, and then enter the gating mechanism. In the gating mechanism, the features are divided into two paths, one of which directly transmits the features, and the other path processes the features through depthwise separable convolution and activation function. The features processed by the depthwise separable convolution and the features of the direct path are multiplied in the gating unit to form a control signal. Then, the control signal is added to the initial input features and the final result is output. Through the gating mechanism, the RGU module can selectively control feature transmission, so that the network can more effectively extract key information at different levels.

7. The TIG welding molten pool image segmentation method based on Mamba and YOLO11 according to claim 6 is characterized in that: The algorithmic representation of the ESSB module includes at least the following steps: Initial convolution, batch normalization, and activation: Input feature F in First, it goes through a convolution layer, batch normalization, and activation function: Among them: BN is batch normalization, As the activation function, the SiLU activation function is used here; the LFA module is designed: Use C3k2 module for preliminary feature extraction: F1=C3k2(F0) Batch Normalization: F2=BN(F1) Non-linear mapping: Residual Connection: F LFA =F0+Conv(F3) The output of the LFA module is subjected to layer normalization, which is expressed as LN: F LN1 =LN(F LFA ) SS2D module: F SS2D =SS2D(F LN1 )+F LN1 Residual connections and weighted fusion: The output of SS2D is weighted and fused with the features of the previous layer: F merge1 =F LN1 +α·F SS2D Among them, α is a learnable weight parameter; Normalize the layer again: F LN2 =LN(F merge1 ) RGU Modules: Initial convolution: X1=Conv(F LN2 ) Feature branches: Branch 1 (direct transfer): X α =X1 Branch 2 (depthwise separable convolution and activation): The input feature X1 is first mixed by 1×1 convolution, then the spatial features are extracted by 3×3 depth-separable convolution, and finally the nonlinear transformation is performed by SiLU activation function, and the output X β : Where: DW_Conv 3·3 It is a 3×3 depth-wise separable convolution; Gating mechanism: Multiply the results of the two branches element-wise: X gated =X α ⊙X β Fusion output and residual connection: F RGU =F LN2 +Conv 1·1 (X gated ) Final output: X out =F RGU 。 8. The TIG welding molten pool image segmentation method based on Mamba and YOLO11 according to claim 7 is characterized in that: The step 3.2 at least includes the following steps: The Simple Stem module is used to quickly downsample the input image in the initial stage and efficiently extract the initial features; The Simple Stem module includes a first layer of convolution and batch normalization and a second layer of convolution and batch normalization; The first layer of convolution and batch normalization is expressed as follows: The second layer of convolution and batch normalization is expressed as follows: The CBS module is used to further enhance feature extraction and fusion in the Neck part, as shown below: N in It is the feature map output after being processed by the ESSB module.

9. The TIG welding molten pool image segmentation method based on Mamba and YOLO11 according to claim 1 is characterized in that: The step 4 at least comprises the following steps: Step 4.1: Convert the model to ONNX format: Export the trained model in PyTorch format to ONNX format to ensure the compatibility of the model between different frameworks; Step 4.2: Use TensorRT to optimize the model. Convert the ONNX model to the TensorRT engine, perform deep optimization for NVIDIA GPUs, and use INT8 quantization technology to further improve the model's reasoning performance, reduce latency, and meet the needs of real-time applications. Step 4.3: Deploy the model to the production environment for comparison.

Images