Steel pipe weld defect detection method based on twinborn VMamba bimodal feature fusion

Abstract

The invention discloses a steel pipe weld defect detection method based on twin VMama bimodal feature fusion, which comprises the following steps: constructing an image data set and a steel pipe weld defect detection model, the model comprising twin coding, feature fusion, decoding and a classifier; the twin coding is composed of a parameter-shared double-branch coding part 1 and a coding part 2, and each coding part is composed of a block division and linear embedding module and a structure unit formed by stacking a plurality of scales layer by layer; the feature fusion part is composed of a cross VMamba feature fusion module; the decoding part is composed of a channel sensing VSS module and a block extension; the training set trains the model, and the test set monitors convergence to obtain a trained model; inputting the verification set into the trained model to obtain a defect detection result; the problems that unobvious defects and pseudo defects are difficult to detect in a single ultrasonic image or an X-ray image, the characteristics of slender and micro crack defects are difficult to detect in a CNN (Convolutional Neural Network)-based method during image processing, and the calculation complexity of a Transform-based method is high are solved.

Description

Technical Field

[0001] This invention belongs to the field of steel pipe weld defect detection technology, specifically relating to a steel pipe weld defect detection method based on twin VMamba dual-modal feature fusion. Background Technology

[0002] Due to the complex and diverse shapes and sizes of welding defects, their uneven distribution and color, as well as the complex background, obstruction, and noise, the detection of welding defects in steel pipes remains a very important and challenging research topic.

[0003] Traditional methods for detecting defects in steel pipe welds suffer from low detection rates and high subjectivity. Currently, ultrasonic imaging can detect internal volumetric defects (such as porosity and slag inclusions) and three-dimensional defects such as incomplete fusion and crack depth in steel pipe welds. However, it has low sensitivity for surface-opening defects (such as shallow cracks) and is greatly affected by the surface roughness of the workpiece. X-ray imaging can detect two-dimensional defects (such as crack direction and porosity distribution) and detect incomplete penetration and linear defects (detection rate 90%–93%), with higher efficiency than ultrasonic testing methods for thin plate welds (<20mm). However, it easily misses cracks parallel to the X-ray direction (such as vertical cracks) and performs poorly on thick plates (>50mm). In image processing, CNN-based methods struggle to capture global context, making it difficult to detect the features of slender, small cracks. While Transformer-based methods can model global relationships, their quadratic computational complexity makes them difficult to process high-resolution weld defect images, limiting their real-time performance and practical applications. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, the present invention aims to propose a steel pipe weld defect detection method based on twin VMamba dual-modal feature fusion, which solves the problems that single ultrasonic or X-ray images are difficult to detect inconspicuous defects and false defects, that CNN-based methods are difficult to detect the features of slender and micro-crack defects during image processing, and that Transformer-based methods have high computational complexity.

[0005] To achieve the above-mentioned objectives, the present invention is implemented through the following technical solution:

[0006] A method for detecting defects in steel pipe welds based on twin VMamba dual-modal feature fusion includes the following steps:

[0007] Step 1: Construct an image pair dataset, wherein the image pair is an ultrasonic image and an X-ray image pair acquired from the same weld seam. The image pair dataset is preprocessed and divided into a training set, a test set and a validation set. The images in the training set are labeled.

[0008] Step 2: Construct a steel pipe weld defect detection model, which consists of a twin coding part, a feature fusion part, a decoding part, and a classifier;

[0009] The twin coding part consists of a parameter-sharing dual-branch coding part 1 and a coding part 2, which are used to extract multi-scale semantic features from the ultrasonic image and X-ray image of the same weld seam, respectively. Each coding part consists of a block partitioning and linear embedding module and multiple scale-stacked structural units. Each structural unit contains two consecutive residual visual state space modules and a block merging module.

[0010] The feature fusion part consists of a cross-VMamba feature fusion module, which is used to align and fuse the multi-scale semantic features of the ultrasonic image and X-ray image of the same weld seam to obtain cross-modal fusion features.

[0011] The decoding part consists of a channel-aware VSS module and a block extension, which is used to decode cross-modal fusion features to obtain features with the same dimension as the input image.

[0012] The classifier is used to output the features with the same dimension as the input image according to the target categories of defective image pixels and non-defective image pixels;

[0013] Step 3: Input the test set, train the steel pipe weld defect detection model, monitor the convergence using the test set, and obtain the optimal model parameters;

[0014] Step 4: Based on the optimal model parameters, input the validation set to obtain the prediction results of steel pipe weld defects.

[0015] Further, in step 2, the block partitioning and linear embedding module partitions and flattens the ultrasonic and X-ray images of the same weld seam into one-dimensional data; the one-dimensional data is input into the structural unit, and multi-scale features with long-distance dependencies are extracted by the residual visual state space module. The multi-scale features are downsampled by the block merging module to obtain multi-scale semantic features of the ultrasonic and X-ray images of the same weld seam.

[0016] Furthermore, the residual visual state space module includes a residual connection structure, a VSS module, and a layer normalization and mapping layer. The one-dimensional data first undergoes layer normalization before entering the VSS module. The VSS module is divided into two branches: the first branch sequentially passes through a linear layer, a depthwise separable convolution, a SiLU activation function, and a two-dimensional selective scan, and then undergoes activation and layer normalization to extract semantic enhancement features; the second branch outputs a feature vector after passing through a linear layer; the outputs of the two branches are multiplied element-wise and then input into a linear layer for feature mixing to obtain the output of the VSS module; the output of the VSS module is then residually connected with the initially input one-dimensional data, and after residual connection, it is processed by a layer normalization and mapping layer to obtain the multi-scale features.

[0017] Furthermore, in step 2, the cross-VMamba feature fusion module consists of a linear layer, a depthwise separable convolutional layer, a ConcatSS layer for fusion, layer normalization, element-wise addition and multiplication operations, a stitching operation, and a scaling operation. The ConcatSS layer transforms the features of the ultrasound image and the X-ray image into corresponding image tensors and sequence tensors in one dimension, respectively, and performs data alignment. It uses element-wise addition and multiplication operations and stitching operations to achieve selective interactive fusion between modalities, and finally forms cross-modal fusion features through linear transformation and residual connection.

[0018] Furthermore, the number of the cross-VMamba feature fusion modules is consistent with the output layer of the twin coding section.

[0019] Furthermore, the specific processing procedure of the cross-VMamba feature fusion module is as follows:

[0020] F′ UI =DWConv(Linear(F UI )),F′ XI =DWConv(Linear(F XI ))

[0021] F Concat =Concat(F′ UI F′ XI )

[0022]

[0023] In the formula, F UI and F XI Multiscale semantic features of ultrasound and X-ray images, respectively, F CVMambaThis represents the cross-modal fused features output by the VMamba feature fusion module. Linear(.) is the feature linearization operation, DWConv(.) is the depthwise separable convolution, Concat(.) is the feature stacking operation, and LN(.) is the layer normalization operation. These are element-wise addition or multiplication operations, respectively. Concat F′ represents the fusion feature after splicing. UI F′ represents the intermediate features extracted from the ultrasound image. XI F″ represents the intermediate features extracted from the X-ray image. UI F″ represents the fusion features of ultrasound images. XI This indicates the fusion features of X-ray images.

[0024] Furthermore, in step 2, the channel-aware VSS module consists of a VSS block, a layer normalization, a convolution, two parallel Avg and Max pooling operations, and two residual connections.

[0025] Furthermore, the first decoding layer in the decoding section contains only block expansion, while the remaining layers include a complete channel-aware VSS module and block expansion.

[0026] Furthermore, the classifier maps the feature map, which has the same dimension as the input image, to two neurons through convolution and outputs the two neurons, which correspond to the two categories of the weld defect image pixel classification task: non-defect pixels and defect pixels.

[0027] Furthermore, in step 3, the binary cross-entropy loss function is used as the objective function, and the model parameters are optimized using the stochastic gradient descent method. The binary cross-entropy loss function is expressed as:

[0028]

[0029] In the formula, GT i,j Pre represents the pixel values ​​of the actual label at positions i and j. i,j Let W represent the pixel values ​​at positions i and j of the image predicted by the model, and let W and H represent the width and height of the image, respectively.

[0030] Compared with the prior art, the beneficial effects of the present invention are:

[0031] This invention utilizes a twin-encoding component to extract features from ultrasonic and X-ray images acquired at the same weld location. The ultrasonic and X-ray images complement each other in weld defect detection, overcoming the limitations of a single image and significantly improving the accuracy of defect detection. A fusion module integrates features from the ultrasonic and X-ray image pairs, and a classifier is used for defect detection. The encoder backbone uses a cascaded VSS module and a downsampling module to extract multi-scale global information from the two modalities. Preliminary feature interaction and fusion are performed through cross-VMamba to enhance cross-modal feature extraction performance and improve the accuracy and efficiency of detection results. VMamba can effectively learn multi-scale detailed feature representations and global context dependencies, capturing long-distance dependencies in weld defect images (such as long fine crack features). It overcomes the difficulty of traditional convolutional neural networks (CNNs) in capturing global context. The linear complexity (O(N)) of VMamba is significantly better than that of Transformer (O(N)). 2 It is especially suitable for processing large-sized steel pipe weld defect images (such as 2048×2048 pixels).

[0032] This invention shares some weights through a twin VMamba architecture, reducing computation by 30% to 40% compared to two independent VMambas. It significantly improves inference efficiency while maintaining accuracy, and has good practical application value. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the steel pipe weld defect detection model of the present invention;

[0034] Figure 2 This is a schematic diagram of the RVM module of the present invention;

[0035] Figure 3 This is a structural diagram of the cross-VMamba feature fusion module of the present invention;

[0036] Figure 4 This is a structural diagram of the Channel Sensing VSS module of the present invention. Detailed Implementation

[0037] The present invention will be further described in detail below through specific embodiments. These embodiments will enable those skilled in the art to have a more comprehensive understanding of the present invention, but will not limit the present invention in any way.

[0038] For the detection of defects in steel pipe welds, ultrasonic images and X-ray images each have their advantages and disadvantages. Using both types of images simultaneously for steel pipe weld defect detection has significant complementary advantages. This invention provides a method for steel pipe weld defect detection based on twin VMamba dual-modal feature fusion, including the following steps:

[0039] Step 1: Image Pair Collection, Augmentation, and Dataset Construction: Acquire ultrasonic and X-ray images of the same defect in a steel pipe weld. Construct a dataset from these image pairs, where each image pair corresponds to the same weld defect. The constructed dataset must include image pairs of steel pipe weld defects under different defect categories and environments to avoid problems such as single images and weak network generalization ability. Augment the images using methods such as image rotation (at angles of 90°, 180°, and 270°), image mirroring (horizontal and vertical), and image edge cropping. Then, pair the images, with each image pair corresponding to the same weld defect. This augments the dataset, compensating for the limited number of image pairs, enhancing the model's generalization ability, and avoiding overfitting.

[0040] Step 2: Construct a steel pipe weld defect detection model. The steel pipe weld defect detection model consists of a twin coding part, a feature fusion part, a decoding part, and a classifier. The twin coding part includes a residual visual state space module (RVM) and a block merging module. The feature fusion part is specifically a cross-VMamba feature fusion module. The decoding part includes a channel-aware VSS module and a block extension model. The classifier is specifically a weld defect detection classifier.

[0041] The architecture of the steel pipe weld defect detection model is as follows: Figure 1 As shown, the specific components and processing steps are as follows:

[0042] Step 2.1: Suppose the input is a pair of ultrasonic and X-ray images of the same weld seam, each image is a two-dimensional image of size H×W×3, input encoding part 1 and encoding part 2 respectively, divide each input image into 16 blocks of size H / 4×W / 4 by block partitioning and linear embedding, then reset to 16 one-dimensional sequences, and adjust the feature size to an arbitrary size by the linear embedding layer, represented as C;

[0043] Step 2.2: Feature extraction is performed using a block merging module with four RVMs × 2 (two consecutive RVMs). Multi-scale features with long-range dependencies are then extracted using the residual visual state space model module, yielding semantic features for both ultrasound and X-ray images. The RVM module structure is as follows: Figure 2 As shown, the feature extraction process of the encoding part is described as follows: After layer normalization, the input is divided into two branches. In the first branch, the input is processed through a linear layer, a depthwise separable convolution, and a 2D selective scan (SS2D) module for feature extraction. After layer normalization, the features are normalized and then multiplied element-wise with the output of the linear layer in the second branch. Finally, the features are mixed using a linear layer, and this result is combined with a residual connection to form the output. Given the input depth features... RVM performs layer normalization, then uses Visual Spectrum Service (VSSM) to capture spatial long-range dependencies, and finally uses an adjustment factor s in the residual connections to achieve better performance. This process is represented as follows:

[0044]

[0045] In the formula, LayerNorm is the layer normalization operation, and s is an adjustment factor with a default value of 0.001.

[0046] The obtained features are transformed into deep features in the mapping layer. This process can be described as follows:

[0047]

[0048] In the formula, Proj represents the mapping operation. F represents the output feature of the l-th layer of the coding part. l This represents the input features of the l-th layer of the coding part, i.e., the output features of the (l-1)-th layer of the coding part.

[0049] exist Figure 2 In RVM, the VSS module is used for remote spatial modeling. Let the input features of the VSS module be... Will Two parallel branches are introduced. In the first branch, the VSS module expands the feature channels using linear layers, followed by Depthwise Separable Convolutional (DWConv), SiLU activation, SS2D, and LayerNorm operations. In the second branch, the VSS module expands the feature channels using linear layers and then uses SiLU activation. Finally, the VSS module aggregates the features from both branches using Hadamard multiplication and projects the channel numbers back to C, generating an output with the same shape as the input features. By introducing depthwise separable convolutional layers into the VSS block, multi-scale feature representations extracted from the encoding part can be captured and aggregated more effectively, resulting in better processing of 2D visual data. The main process is described as follows:

[0050] In the formula, This represents the Hadamard multiplication operation, G l G represents the input features of the l-th layer, and G1 and G2 represent the intermediate features after processing by the two branches, respectively. out This indicates the output characteristics of the VSS module.

[0051] The scaling factor is mainly used for feature normalization. It dynamically scales the feature output after selective spatial modeling (SS2D) to avoid feature instability such as gradient explosion and vanishing, and provides adaptive adjustment of channel or spatial dimensions. It has the functions of suppressing irrelevant regions and enhancing key features.

[0052] Step 2.3: The cross-VMamba feature fusion module is inserted layer by layer at each scale level, with its number consistent with the encoder output layer, totaling 4 groups, for aligning and fusing bimodal features at multiple scales. Its structure is as follows: Figure 3 The diagram shows a system composed of linear layers, depthwise separable convolutional layers, ConcatSS layers for fusion, layer normalization, element-wise addition, multiplication, and scaling operations. The cross-VMamba feature fusion module utilizes a selective scanning mechanism (S6) to establish dynamic spatial channel attention associations between bimodal features, outputting a cross-modal fused feature map. The ConcatSS layer transforms the features of the ultrasound and X-ray images into corresponding image tensors and sequence tensors, respectively, performs data alignment, and then fuses them using a selected feature fusion strategy to obtain the bimodal features. (Refer to...) Figure 3 The cross-VMamba feature fusion process is described as follows:

[0053] F′ UI =DWConv(Linear(F UI )),F′ XI =DWConv(Linear(F XI ))

[0054] F Concat =Concat(F′ UI F′ XI )

[0055]

[0056] In the formula, F UI and F XI Semantic features of ultrasound images and X-ray images, respectively, F CVMamba This is the output of the cross-VMamba feature fusion module. DWConv(.) is a depthwise separable convolution, Concat(.) is a feature stacking operation, and LN(.) is a layer normalization operation. These are element-wise addition and multiplication operations, respectively. Concat F′ represents the fusion feature after splicing. UI F′ represents the intermediate features extracted from the ultrasound image. XI F″ represents the intermediate features extracted from the X-ray image. UI F″ represents the fusion features of ultrasound images. XI This indicates the fusion features of X-ray images.

[0057] Step 2.4: To improve the feature recovery capability of the decoding section, channel-aware VSS modules are inserted layer by layer into the decoding stage structure. The first decoding layer only contains block expansion operations, while the remaining layers use the complete channel-aware VSS module structure. The channel-aware VSS module structure is as follows: Figure 4 As shown, it consists of a VSS block, a layer normalization, a convolution, two parallel Avg and Max pooling operations, and two residual connections. The channel-aware VSS module of the decoding part receives the cross-modal fusion feature map, progressively upsamples the cross-modal fusion feature map and restores the spatial dimension, while performing precise defect localization. Linear projection is used to aggregate multi-scale channel dimension features and expand them by 16 times. Then, depthwise separable convolution is used to aggregate spatial dimension features, expanding the feature resolution of the aggregated channel and spatial dimension results to the size of the input image. The process is described as follows:

[0058]

[0059] F MAP =Sigmoid(Concat(Maxpool(F VSSC ),Avgpool(F VSSC )))

[0060] F VSSC =Conv(LN(VSS(F CVMamba )))

[0061] In the formula, F CAVSS The output of the channel-aware VSS module is defined by VSS(·), which is the VSS operation Sigmoid(.) activation operation, Conv(.) convolution operation, Maxpool(.) and Avgpool(.) are the max pooling and average pooling operations, respectively.

[0062] Step 2.5: The classifier maps the expanded feature map to two neurons through convolution and outputs the result. The two neurons correspond to the two categories of the weld defect image pixel classification task, namely, non-defect pixels and defect pixels.

[0063] Step 3: The method for training the twin VMamba bimodal feature fusion model, in the training phase, uses binary cross-entropy as the objective function, assuming Pre i Let GT represent the image predicted by the i-th model. i Let represent the Ground Truth of the i-th image. Then, the binary cross-entropy loss function is expressed as:

[0064]

[0065] In the formula, GTi,j Represents the pixel values ​​at positions i and j of the Ground Truth, Pre i,j Let W represent the pixel values ​​at positions i and j of the image predicted by the model, and let W and H represent the width and height of the image, respectively.

[0066] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above, which are merely illustrative and not restrictive. Those skilled in the art, upon learning of the present invention, can make various modifications to these described embodiments without departing from the spirit and scope of the claims, and all such modifications fall within the scope of protection of the present invention. Parts of the present invention not described in detail are well-known to those skilled in the art.

Claims

1. A method for detecting defects in steel pipe welds based on twin VMamba dual-modal feature fusion, characterized in that, Includes the following steps: Step 1: Construct an image pair dataset, wherein the image pair is an ultrasonic image and an X-ray image pair acquired from the same weld seam. The image pair dataset is preprocessed and divided into a training set, a test set and a validation set. The images in the training set are labeled. Step 2: Construct a steel pipe weld defect detection model, which consists of a twin coding part, a feature fusion part, a decoding part, and a classifier; The twin coding part consists of a parameter-sharing dual-branch coding part 1 and a coding part 2, which are used to extract multi-scale semantic features from the ultrasonic image and X-ray image of the same weld seam, respectively. Each coding part consists of a block partitioning and linear embedding module and multiple scale-stacked structural units. Each structural unit contains two consecutive residual visual state space modules and a block merging module. The feature fusion part consists of a cross-VMamba feature fusion module, which is used to align and fuse the multi-scale semantic features of the ultrasonic image and X-ray image of the same weld seam to obtain cross-modal fusion features. The decoding part consists of a channel-aware VSS module and a block extension, which is used to decode cross-modal fusion features to obtain features with the same dimension as the input image. The classifier is used to output the features with the same dimension as the input image according to the target categories of defective image pixels and non-defective image pixels; Step 3: Input the test set, train the steel pipe weld defect detection model, monitor the convergence using the test set, and obtain the optimal model parameters; Step 4: Based on the optimal model parameters, input the validation set to obtain the prediction results of steel pipe weld defects.

2. The method for detecting defects in steel pipe welds based on twin VMamba dual-modal feature fusion according to claim 1, characterized in that, In step 2, the block partitioning and linear embedding module partitions and flattens the ultrasonic and X-ray images of the same weld seam into one-dimensional data. The one-dimensional data is input into the structural unit, and multi-scale features with long-distance dependencies are extracted by the residual visual state space module. The multi-scale features are downsampled by the block merging module to obtain multi-scale semantic features of the ultrasonic and X-ray images of the same weld seam.

3. The method for detecting defects in steel pipe welds based on twin VMamba dual-modal feature fusion according to claim 2, characterized in that, The residual visual state space module includes a residual connection structure, a VSS module, and a layer normalization and mapping layer. The one-dimensional data first undergoes layer normalization processing before entering the VSS module. The VSS module is divided into two branches: the first branch sequentially passes through a linear layer, a depthwise separable convolution, a SiLU activation function, and a two-dimensional selective scan, and then undergoes activation and layer normalization to extract semantic enhancement features. The second branch outputs a feature vector after passing through a linear layer; the outputs of the two branches are multiplied element-wise and then input into a linear layer for feature mixing to obtain the output of the VSS module; the output of the VSS module is then residually concatenated with the initial input one-dimensional data, and after the residual concatenation, it is processed by layer normalization and mapping layer to obtain the multi-scale features.

4. The method for detecting defects in steel pipe welds based on twin VMamba dual-modal feature fusion according to claim 1, characterized in that, In step 2, the cross-VMamba feature fusion module consists of a linear layer, a depthwise separable convolutional layer, a ConcatSS layer for fusion, layer normalization, element-wise addition and multiplication operations, a stitching operation, and a scaling operation. The ConcatSS layer transforms the features of the ultrasound image and X-ray image into corresponding image tensors and sequence tensors in one dimension, respectively, and performs data alignment. It uses element-wise addition and multiplication operations and stitching operations to achieve selective interactive fusion between modalities, and finally forms cross-modal fusion features through linear transformation and residual connection.

5. The method for detecting defects in steel pipe welds based on twin VMamba dual-modal feature fusion according to claim 4, characterized in that, The number of cross-VMamba feature fusion modules is consistent with the output layer of the twin coding part.

6. The method for detecting defects in steel pipe welds based on twin VMamba dual-modal feature fusion according to claim 4, characterized in that, The specific processing procedure of the cross-VMamba feature fusion module is as follows: F' UI =DWConv(Linear(F UI )),F' XI =DWConv(Linear(F XI )) F Concat =Concat(F' UI ,F' XI ) In the formula, F UI and F XI Multiscale semantic features of ultrasound and X-ray images, respectively, F CVMamba This represents the cross-modal fused features output by the VMamba feature fusion module. Linear(.) is the feature linearization operation, DWConv(.) is the depthwise separable convolution, Concat(.) is the feature stacking operation, and LN(.) is the layer normalization operation. These are element-wise addition or multiplication operations, respectively. Concat F' represents the blended feature after splicing. UI F' represents the intermediate features extracted from the ultrasound image. XI F represents the intermediate features extracted from the X-ray image. UI F” represents the fusion features of ultrasound images. XI This indicates the fusion features of X-ray images.

7. The method for detecting defects in steel pipe welds based on twin VMamba dual-modal feature fusion according to claim 1, characterized in that, In step 2, the channel-aware VSS module consists of a VSS block, a layer normalization, a convolution, two parallel Avg and Max pooling operations, and two residual connections.

8. The method for detecting defects in steel pipe welds based on twin VMamba dual-modal feature fusion according to claim 1, characterized in that, The first decoding layer in the decoding section contains only block expansion, while the remaining layers include a complete channel-aware VSS module and block expansion.

9. The method for detecting defects in steel pipe welds based on twin VMamba dual-modal feature fusion according to claim 1, characterized in that, The classifier maps the feature map, which has the same dimension as the input image, to two neurons through convolution and outputs the two neurons, which correspond to the two categories of the weld defect image pixel classification task: non-defect pixels and defect pixels.

10. The method for detecting defects in steel pipe welds based on twin VMamba dual-modal feature fusion according to claim 1, characterized in that, In step 3, the binary cross-entropy loss function is used as the objective function, and the model parameters are optimized using the stochastic gradient descent method. The binary cross-entropy loss function is expressed as: In the formula, GT i,j Pre represents the pixel values ​​of the actual label at positions i and j. i,j Let W represent the pixel values ​​at positions i and j of the image predicted by the model, and let W and H represent the width and height of the image, respectively.

Images