Friction stir welding seam real-time defect detection method based on multi-stage context polymerization and knowledge distillation

Abstract

The invention provides a friction stir welding seam real-time defect detection method based on multistage context aggregation and knowledge distillation, and belongs to the technical field of welding seam defect detection.The method comprises the following steps that a friction stir welding seam image data set is constructed, and a constructed training set, a constructed verification set and a constructed test set are subjected to preprocessing operation; establishing a friction stir welding defect detection model based on ISIA-HKD, training the constructed defect detection model by using the pre-processed training set, checking a training result by using the pre-processed verification set to obtain an intermediate training model, evaluating the trained intermediate model, selecting the model with the best evaluation performance as a final model, and finally, carrying out defect detection on the friction stir welding defect. And deploying the final model. Local textures and global structural features of defects in a welding image are effectively captured through a multi-stage context aggregation module, noise is suppressed in combination with a gating mechanism, a defect area is highlighted, and the tiny defect detection capacity is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of weld defect detection technology, and in particular to a real-time defect detection method for friction stir welds based on multi-level context aggregation and knowledge distillation. Background Technology

[0002] Friction stir welding is a critical process in aerospace, rail transportation, and other fields, and the quality of its welds directly affects structural safety. Traditional manual visual inspection, offline inspection techniques, and machine vision methods based on hand-crafted features suffer from low efficiency, high false negative rates, and difficulty adapting to real-time inspection requirements. Furthermore, welded images generally exhibit low texture, high noise, and blurred edges, making defect feature extraction difficult. Meanwhile, deep learning models face challenges in industrial applications: high-precision models are computationally complex and difficult to deploy on resource-constrained edge devices, while lightweight models suffer from insufficient detection accuracy due to their small parameter size. Publicly available welding defect datasets are also scarce, limiting model generalization capabilities. Existing knowledge distillation methods often focus on feature transfer in the output layer or backbone network, neglecting the deep semantic information of the feature pyramid network, making it difficult to effectively handle the multi-scale detection requirements of welding defects. Therefore, a method integrating contextual modeling and multi-level knowledge distillation is urgently needed to improve the real-time performance, accuracy, and generalization ability of welding defect detection. Summary of the Invention

[0003] The purpose of this invention is to provide a real-time defect detection method for friction stir welding seams based on multi-level context aggregation and knowledge distillation. This method addresses the technical problem that existing knowledge distillation methods often focus on feature transfer in the output layer or backbone network, neglecting the deep semantic information of the feature pyramid network, thus failing to effectively handle the multi-scale detection requirements of welding defects. It can improve the real-time performance, accuracy, and generalization ability of welding defect detection.

[0004] This paper introduces the ISIA-HKD weld defect detection model. This method effectively captures the local texture and global structural features of defects in welding images through a multi-level context aggregation module. Combined with a gating mechanism, it suppresses noise and highlights defect areas, significantly improving the detection capability of minute defects. Simultaneously, it utilizes a multi-level knowledge distillation strategy to transfer knowledge from a large model to a lightweight model, maintaining high accuracy while compressing parameter scale, enabling the model to achieve good real-time inference speed on industrial edge devices. Furthermore, its pluggable modular design is compatible with mainstream detection frameworks. Combined with dedicated datasets and data augmentation techniques, it enhances generalization ability and can provide real-time feedback of detection results to drive welding process parameter adjustments, reducing manual inspection costs and equipment wear. Compared with traditional methods, it achieves comprehensive breakthroughs in detection accuracy, real-time performance, model efficiency, and industrial adaptability, providing an efficient and reliable solution for intelligent quality control of friction stir welding.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A real-time defect detection method for friction stir welds based on multi-level context aggregation and knowledge distillation, the method comprising the following steps:

[0007] Step 1: Construct a dataset of images of defects in friction stir welding;

[0008] Step 2: Perform preprocessing operations on the constructed training set, validation set, and test set;

[0009] Step 3: Establish a friction stir welding defect detection model based on ISIA-HKD;

[0010] Step 4: Train the constructed defect detection model using the preprocessed training set, and verify the training results using the preprocessed validation set to obtain an intermediate training model;

[0011] Step 5: Evaluate the intermediate trained models. The model evaluation and deployment use a test set to evaluate candidate models, focusing on detection accuracy, false detection rate and inference speed. Select the model with the best evaluation performance as the final model.

[0012] Step 6: Deploy the final model to the Jetson Xavier NX embedded device and integrate it into the welding robot real-time acquisition system to perform online detection of infrared or X-ray images of the welding process on the production line, provide real-time feedback on defect type and location, and drive dynamic adjustment of process parameters.

[0013] Further, the specific process of step 1 is as follows: acquire friction stir welding defect video data with known defect locations and types, segment the video data into image frames, and classify them according to different process parameters, different defect types, and different image features to obtain an image dataset that can be used for network training. After constructing the dataset, further expand the dataset and divide the expanded dataset into training set, validation set, and test set according to the proportion.

[0014] Furthermore, the preprocessing in step 2 includes multimodal data fusion, metal texture enhancement, and adaptive image normalization.

[0015] Furthermore, in step 3, the friction stir welding defect detection model includes an input end, a context aggregation module, a multi-level knowledge distillation strategy, and an output end.

[0016] Furthermore, the context aggregation module includes a hierarchical convolution module, a dynamic weighted fusion module, and a residual optimization module, which are responsible for extracting the image features fed from the input end;

[0017] The hierarchical convolution module adopts a three-level depthwise separable convolutional cascade structure, with each layer followed by a GeLU activation function. By progressively expanding the receptive field, it extracts weld features from local texture to global structure in layers. The weld features include small convolutional kernels to capture the details of microcrack edges and large convolutional kernels to model the heat diffusion morphology of incomplete weld penetration areas. Finally, global average pooling is used to enhance the distinction between defects and background, significantly improving the ability to express multi-scale features.

[0018] The dynamic weighted fusion module generates a spatial hierarchical perceptual weight tensor based on the original features. It then uses element-level weighted adaptive fusion of a four-layer feature pyramid. The gating mechanism enables the network to autonomously allocate weights and actively reduce weights in noisy areas such as metallic reflections, achieving a precise balance between enhancing defective areas and suppressing the background.

[0019] The residual optimization module first performs channel compression on the fused features using lightweight convolutional blocks, integrates key information across channels, and then adds the optimization results to the original features through residual connections. This not only protects high-frequency texture details and avoids smoothing of deep networks, but also accelerates gradient convergence, comprehensively optimizing the robustness of welding defect localization.

[0020] Furthermore, the multi-level knowledge distillation strategy employs a three-level alignment strategy. The feature alignment layer uses L2 loss to constrain the student model to mimic the multi-level feature maps of the teacher model. The correlation distillation layer calculates channel similarity weights, focusing on key cross-modal features. The mutual information optimization layer improves the discriminative power for complex defects through probability distribution matching.

[0021] Furthermore, the distillation fusion network for the multi-level knowledge distillation strategy adopts the lightweight YOLO11-N architecture of the YOLO series as the student model and the large-scale YOLO11-L model as the teacher model, which is responsible for basic feature extraction. The detection head, based on the enhanced feature map, outputs the target's location, category, and confidence score. A single-stage detection head design is adopted to ensure real-time performance.

[0022] Furthermore, the output uses mutual information loss as the loss function to align the deep statistical features of the decoder and solve the problem of complex scale relationships. It approximates the probability distribution through a probability simulation module, which includes a probability feature extractor and a probability generator. It uses attention mechanism and linear layer to generate probability feature vectors. Finally, it optimizes through mutual information loss function to achieve cross-level knowledge transfer from shallow to deep layers.

[0023] Furthermore, in step 4, training is based on minimizing the multi-stage distillation loss function, where the hyperparameters are dynamically adjusted through the validation set during the training process.

[0024] The model deployment module is based on the Jetson industrial-grade edge computing module, which deploys the ISIA-HKD model to embedded devices to perform online detection of infrared / X-ray images acquired in real time during the welding process. Through lightweight design and multi-level knowledge distillation strategy, it maintains high accuracy while compressing the model parameter scale, meets the real-time detection needs of the production line, and realizes intelligent and automated monitoring of welding quality.

[0025] The present invention, by adopting the above-described technical solution, has the following beneficial effects:

[0026] This invention effectively captures the local texture and global structural features of defects in welding images through a multi-level context aggregation module. Combined with a gating mechanism, it suppresses noise and highlights defect areas, significantly improving the detection capability of minute defects. Simultaneously, it utilizes a multi-level knowledge distillation strategy to transfer knowledge from a large model to a lightweight model, maintaining high accuracy while compressing parameter scale, enabling the model to achieve good real-time inference speed on industrial edge devices. Furthermore, its pluggable module design is compatible with mainstream detection frameworks, and its generalization ability is enhanced by combining dedicated datasets and data augmentation techniques. It can also provide real-time feedback on detection results to drive adjustments to welding process parameters, reducing manual inspection costs and equipment wear. This method significantly improves defect detection accuracy, reduces false detection rates, and its inference speed meets the real-time requirements of production lines. Moreover, the model can be deployed on embedded devices, greatly improving welding quality inspection efficiency and reducing maintenance costs, demonstrating strong practical value for ensuring the safety of high-end equipment manufacturing. Attached Figure Description

[0027] Figure 1 This is a flowchart of the method of the present invention;

[0028] Figure 2 This is a diagram of the friction stir welding defect detection model based on ISIA-HKD of the present invention;

[0029] Figure 3 This is a structural diagram of the ISIA module of the present invention;

[0030] Figure 4 This is a structural diagram of the probability simulation module of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and preferred embodiments. However, it should be noted that many details listed in the specification are merely to provide the reader with a thorough understanding of one or more aspects of the present invention, and these aspects of the invention can be implemented even without these specific details.

[0032] like Figure 1As shown, the real-time defect detection method for friction stir welds based on multi-level context aggregation and knowledge distillation includes the following steps:

[0033] S101, Construct a dataset of images showing defects in friction stir welding.

[0034] The dataset in this example originates from a friction stir welding process video captured by the inventor using an industrial camera. Image frames containing defect types such as holes and cracks were extracted from the video. The images were classified according to different process parameters (such as stirring head rotation speed and welding speed), defect types, and image features (such as heat distribution and texture). The LabelImg tool was used to label the defect locations and types, resulting in the original dataset. Data augmentation methods such as geometric transformations (rotation and scaling) and contrast enhancement were used to increase sample diversity. The augmented dataset was then divided into training, validation, and test sets in a 7:2:1 ratio.

[0035] S102, Establish a friction stir welding defect detection model based on ISIA-HKD.

[0036] The flow details of the friction stir welding defect image data in the model are shown in the appendix. Figure 2 The model includes an input end, a context aggregation module, a multi-level knowledge distillation strategy module, and an output end.

[0037] The context aggregation module includes a hierarchical convolution module, a dynamic weighted fusion module, and a residual optimization module. The specific process is as follows:

[0038] Hierarchical Convolutional Module: After standardization, the input image enters a cascaded structure of three levels of depthwise separable convolutions (3×3, 5×5, and 7×7 kernels), with each layer followed by a GeLU activation function. Small convolutional kernels capture local details such as micro-crack edges, while large convolutional kernels model the global heat diffusion morphology of incomplete weld areas. Finally, global average pooling enhances the distinction between defects and the background. Specifically, the initial feature map F... 0 It is projected into a new feature space through a linear transformation: F 1 =Linear(F 0 For each level 1∈(1,3), the effective receptive field is significantly larger than the kernel size k. 1 Furthermore, the final feature maps are processed using a global average pooling layer to capture the global context of the entire input. Finally, we obtain a set of output multi-scale feature maps. (1 indicates a level), this collection captures both short-range and long-range contexts at different granularity levels.

[0039] F l =GeLU(DWConv(F l-1 ))

[0040] Dynamic Weighted Fusion Module: The detection model utilizes local fine-grained features to encode queries for salient visual objects, while queries for background scenes primarily rely on global coarse-grained features. To fully leverage contextual information at different granularities, the model introduces a gating mechanism to control the amount of information aggregated from different levels for each query. Specifically, the spatial and hierarchical awareness gating weights are first calculated using the formula W. gate =Linear(F 0 The spatial-hierarchical perceptual weight tensor is generated based on the original feature map. A four-layer feature pyramid (including the original input features) is adaptively fused using element-wise weighting. A gating mechanism assigns higher weights to defect areas such as cracks and lowers the weights to areas with metallic reflective noise. Then, element-wise multiplication is used to perform weighted summation to obtain the output feature map, as shown below: Output fused features

[0041]

[0042] Residual optimization module: To enable communication between different channels, the feature maps undergo a series of convolutional operations (Conv()), normalization layers (LN()), and corrected linear units (ReLU()), instead of a simple linear mapping. Finally, a convolutional layer optimizes the output to ensure alignment with the original features. The entire process can be described as follows: F out′ =Conv(ReLU(LN(Conv(F) out A lightweight convolutional block (1×1 convolution → LayerNorm → ReLU) is used to perform channel compression (reducing the number of channels to 1 / 16) on the fused features. Key information such as the correlation between holes and heat distribution is integrated across channels. The optimized feature map F is then added to the original features through residual connections. opt It preserves high-frequency texture details and accelerates gradient convergence.

[0043] The multi-level knowledge distillation strategy module adopts a teacher-student architecture, with the teacher model being YOLO11-L and the student model being YOLO11-N. Specifically, it includes a three-level alignment strategy:

[0044] Feature alignment layer: This layer uses L2 loss to constrain the student model to mimic the teacher model's multi-level feature maps. The loss function is... in These are feature maps for the student and teacher models, respectively.

[0045] Correlation distillation layer: Channel similarity weights are calculated, and key cross-modal features are focused using spatial-channel weighted L2 loss. To further alleviate the computationally intensive spatial average pooling layer problem, the P_s() function is applied to each channel to reduce vector dimensionality. The entire process can be described as follows:

[0046]

[0047] Similar to the channel-related weights, the inner product of the student and teacher model vectors at their corresponding spatial locations is calculated. These values ​​are then converted into probability vectors using the softmax function. Spatial-related weights W spatial The formula is as follows:

[0048]

[0049] First, the feature map is convolved with a 1×1 layer to adjust its dimensions. After spatial average pooling, the inner product is calculated, and channel weights are generated using softmax. Spatial weights The loss function is

[0050]

[0051] Mutual Information Optimization Layer: Multi-scale, high-level semantic information is ultimately input into the decoder of the detection network to predict the final detection result. However, direct feature alignment methods, similar to those used in encoders, often struggle to effectively handle complex scale relationships, resulting in an inability to fully utilize semantically rich feature maps for accurate prediction. In contrast, we align the deep statistical representations between the student decoder and the teacher decoder using a mutual information model I(·), as follows:

[0052]

[0053] However, accurately calculating the joint probability and marginal probability between the two decoders is impractical. Therefore, we employ a neural network model to approximate these probabilities. First, the feature maps of the two models are concatenated along the channel dimension to represent their joint probability. Next, these feature maps are combined with zero matrices of the same size, modeling them separately as independent variables. Then, a set of Probabilistic Simulation Modules (PSMs) approximates the joint probability distribution using these combined representations. The PSM consists of two key components: a Probabilistic Feature Extractor (PFE) and a Probabilistic Generator (PG). The structure of the PSM is as follows: Figure 4 As shown.

[0054] The joint probability distribution of the teacher-student model feature maps is approximated through a probability simulation module, which includes a probabilistic feature extractor and a probability generator. The Probabilistic Feature Extractor (PFE) generates probabilistic feature vectors using an attention mechanism, while the Probabilistic Generator (PG) simulates the probability distribution through linear layers and activation functions. Finally, the mutual information loss formula is used:

[0055]

[0056] Align deep statistical features to improve the ability to detect complex defects.

[0057] To reproduce the PFE component, an attention mechanism was employed. Specifically, we first vectorized each channel, then obtained the query matrix Q and key matrix K through a linear layer. Next, we calculated the attention weights by scaling the dot product, and then applied the softmax function σ(.) to generate the probabilistic feature vector. The entire process can be described as follows:

[0058]

[0059] Here, Vp represents the probabilistic feature vector, and d represents the length of the feature vector. Furthermore, these probabilistic feature vectors are input into a PG component consisting of multiple Probabilistic Simulation Units (PSUs). Each PSU computes the attention weight Watt in the probabilistic feature vector through a linear layer, effectively focusing more attention on more important locations. This operation can be represented as:

[0060] V′ p =σ(W att ·V p )·V p

[0061] Among them, V′ p This represents the weighted input vector. This weighted vector is then further processed through multiple LLR (linear + layer normalization + ReLU) units to extract features. To prevent overfitting, Dropout is applied in each LLR unit. Finally, a linear layer and a sigmoid function are used to generate the final probability output. In summary, the PG component helps approximate the joint probability and marginal probability, thus calculating the mutual information value. According to consensus, higher mutual information values ​​indicate a stronger relationship between the feature maps. To narrow the correlation gap between the two decoders, we introduce a mutual information alignment loss function. Defined as:

[0062]

[0063] The output employs a mutual information loss function to optimize the detection head, outputting the target location, category, and confidence score based on the enhanced feature map. The detection head is a single-stage design, using CIoU loss to calculate the difference between the predicted and ground truth bounding boxes, and non-maximum suppression (NMS) to eliminate redundant detection results, outputting the defect category and localization result.

[0064] S103, Model Training and Validation: The training set is read in the experimental environment, based on minimizing the total loss. (α=1×10 -3 β=1×10 -8 γ = 1 × 10 -6Multiple training sessions were conducted. After each training round, the hyperparameters were adjusted using the validation set, the gradient of the teacher model was frozen, and the student model was optimized. During the training process, metrics such as AP50:95 and FPS were monitored, and the intermediate model with the best performance on the validation set was selected as a candidate.

[0065] S104. In the experimental environment, read the prepared training dataset, and perform multi-batch, multi-round training using the corresponding deep learning development platform based on minimizing the following loss. During the training process, the validation set is used to evaluate the model performance. The model with the best performance during the training process is selected as the result model. The final model is deployed to the JetsonXavierNX embedded device and integrated into the welding robot real-time acquisition system to perform online detection of infrared / X-ray images of the welding process on the production line, provide real-time feedback on defect categories and locations, and drive dynamic adjustment of process parameters.

[0066] This embodiment relies on the following conditions, as shown in Table 1, for training experiments.

[0067] Table 1 shows the training experiment data.

[0068]

[0069] S105 evaluates the intermediate models trained in S104. Model evaluation and deployment use a test set to evaluate candidate models, focusing on detection accuracy (AP), false positive rate (FPS), and inference speed (FPS). The model with the best evaluation performance is selected as the final model. To accelerate network convergence, this example uses cross-validation during training to evaluate intermediate models. Specifically, after every 200 training iterations, the entire model is frozen, validated for 50 training iterations, and then unfrozen, repeating this process until 5000 training iterations are completed. Since the image size in the dataset is fixed, requiring significant GPU memory, this example sets the batch size for each training iteration to 8.

[0070] S106, Apply the final model selected in S105 to a complex real-world scenario. Specifically, the final model is considered as a friction stir welding defect detection method based on ISIA-HKD proposed in this invention. This model is deployed into a Jetson Xavier NX embedded device and integrated into the welding robot's real-time acquisition system to perform online detection of infrared / X-ray images of the welding process on the production line, providing real-time feedback on defect type and location, and driving dynamic adjustment of process parameters.

[0071] Matters not covered in this invention are common knowledge.

[0072] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A real-time defect detection method for friction stir welds based on multi-level context aggregation and knowledge distillation, characterized in that, The method includes the following steps: Step 1: Construct a dataset of images of defects in friction stir welding; Step 2: Perform preprocessing operations on the constructed training set, validation set, and test set; Step 3: Establish a friction stir welding defect detection model based on ISIA-HKD; Step 4: Train the constructed defect detection model using the preprocessed training set, and verify the training results using the preprocessed validation set to obtain an intermediate training model; Step 5: Evaluate the intermediate trained models. The model evaluation and deployment use a test set to evaluate candidate models, focusing on detection accuracy, false detection rate and inference speed. Select the model with the best evaluation performance as the final model. Step 6: Deploy the final model to the Jetson Xavier NX embedded device and integrate it into the welding robot real-time acquisition system to perform online detection of infrared or X-ray images of the welding process on the production line, provide real-time feedback on defect type and location, and drive dynamic adjustment of process parameters.

2. The real-time defect detection method for friction stir welds based on multi-level context aggregation and knowledge distillation according to claim 1, characterized in that: The specific process of step 1 is as follows: acquire video data of friction stir welding defects with known defect locations and types, segment the video data into image frames, and classify them according to different process parameters, different defect types, and different image features to obtain an image dataset that can be used for network training. After constructing the dataset, further expand the dataset and divide the expanded dataset into training set, validation set, and test set according to the proportion.

3. The real-time defect detection method for friction stir welds based on multi-level context aggregation and knowledge distillation according to claim 1, characterized in that: The preprocessing in step 2 includes multimodal data fusion, metallic texture enhancement, and adaptive image normalization.

4. The real-time defect detection method for friction stir welds based on multi-level context aggregation and knowledge distillation according to claim 1, characterized in that: In step 3, the friction stir welding defect detection model includes an input end, a context aggregation module, a multi-level knowledge distillation strategy, and an output end.

5. The real-time defect detection method for friction stir welds based on multi-level context aggregation and knowledge distillation according to claim 4, characterized in that: The context aggregation module includes a hierarchical convolution module, a dynamic weighted fusion module, and a residual optimization module, which are responsible for extracting image features fed from the input end; The hierarchical convolution module adopts a three-level depthwise separable convolutional cascade structure, with each layer followed by a GeLU activation function. By progressively expanding the receptive field, it extracts weld features from local texture to global structure in layers. The weld features include small convolutional kernels to capture the details of microcrack edges and large convolutional kernels to model the heat diffusion morphology of incomplete weld penetration areas. Finally, global average pooling is used to enhance the distinction between defects and background, significantly improving the ability to express multi-scale features. The dynamic weighted fusion module generates a spatial hierarchical perceptual weight tensor based on the original features. It then uses element-level weighted adaptive fusion of a four-layer feature pyramid. The gating mechanism enables the network to autonomously allocate weights and actively reduce weights in noisy areas such as metallic reflections, achieving a precise balance between enhancing defective areas and suppressing the background. The residual optimization module first performs channel compression on the fused features using lightweight convolutional blocks, integrates key information across channels, and then adds the optimization results to the original features through residual connections. This not only protects high-frequency texture details and avoids smoothing of deep networks, but also accelerates gradient convergence, comprehensively optimizing the robustness of welding defect localization.

6. The real-time defect detection method for friction stir welds based on multi-level context aggregation and knowledge distillation according to claim 4, characterized in that: The multi-level knowledge distillation strategy adopts a three-level alignment strategy. The feature alignment layer uses L2 loss to constrain the student model to imitate the multi-level feature map of the teacher model. The correlation distillation layer calculates the channel similarity weights and focuses on key cross-modal features. The mutual information optimization layer improves the discrimination power of complex defects through probability distribution matching.

7. The real-time defect detection method for friction stir welds based on multi-level context aggregation and knowledge distillation according to claim 5, characterized in that: The distillation fusion network of the multi-level knowledge distillation strategy adopts the lightweight architecture YOLO11-N of the YOLO series as the student model and the large model YOLO11-L as the teacher model, which is responsible for basic feature extraction. The detection head is based on the enhanced feature map and outputs the location, category and confidence of the target. A single-stage detection head design is adopted to ensure real-time performance.

8. The real-time defect detection method for friction stir welds based on multi-level context aggregation and knowledge distillation according to claim 4, characterized in that: The output uses mutual information loss as the loss function to align the deep statistical features of the decoder and solve the problem of complex scale relationships. It approximates the probability distribution through a probability simulation module, which includes a probability feature extractor and a probability generator. It uses attention mechanism and linear layer to generate probability feature vectors and finally optimizes them through mutual information loss function to achieve cross-level knowledge transfer from shallow to deep layers.

9. The real-time defect detection method for friction stir welds based on multi-level context aggregation and knowledge distillation according to claim 4, characterized in that: In step 4, training is based on minimizing the multi-stage distillation loss function, where the hyperparameters are dynamically adjusted using a validation set during the training process.

Images