Real-time domain adaptive defect detection method based on double alignment and uncertainty filtering

Abstract

The invention discloses a real-time domain adaptive defect detection method based on double alignment and uncertainty filtering. The method comprises the following steps: firstly, constructing a defect detection data set which comprises source domain data and target domain data; then, constructing a defect detection model which comprises a backbone network, an encoder, a decoder and a detection head; and finally, pre-training the defect detection model by using the source domain data to obtain a teacher model, and storing the multi-scale source domain features extracted by the teacher model backbone network into a feature database in groups according to scales. Initializing the defect detection model by using the teacher model parameters to obtain a student model; and carrying out collaborative optimization on the teacher model and the student model by using a pseudo-bounding box and feature distribution double-alignment strategy to realize real-time domain adaptive detection. According to the method, performance degradation caused by domain offset is effectively relieved through a double-alignment strategy, meanwhile, error tag accumulation is avoided through uncertainty perception and filtering of a pseudo-bounding box, and the stability during domain adaptation is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of industrial defect detection, and specifically relates to a real-time domain adaptive defect detection method based on double alignment and uncertainty filtering. BACKGROUND

[0002] With the development of high-precision technology, the quality requirements for industrial precision components such as gas turbine blades, wind turbine blades and aircraft engine blades are becoming higher and higher, and surface defects are the key factors affecting the quality of the components, and can cause failure and safety accidents in severe cases. For example, the aircraft engine blade is the core component of the aircraft power system, and surface defects such as ablation, fracture, crack, groove and depression will quickly deteriorate under the extreme high temperature and high pressure working conditions during engine operation, which may cause catastrophic failure, so defect detection is a key link in blade manufacturing and maintenance.

[0003] In recent years, computer vision methods based on deep learning have become the mainstream means of industrial defect detection. By training the model with a large number of labeled images, the model learns the defect features, realizes defect positioning and classification, and has higher accuracy and efficiency compared with manual detection. However, the source domain data used for model training is usually from a laboratory scene with fixed light and shooting angle, while the target domain data for actual application and testing is usually from a factory scene with dynamic light and multiple perspectives. The feature distribution of the source domain and the target domain data is inconsistent, and there is a core problem of "domain shift", which further leads to a significant decrease in detection accuracy. To alleviate the domain shift, traditional domain adaptation (DA) technology needs to process the target domain data offline in batches, which cannot meet the needs of the factory "real-time detection-sorting" pipeline operation. Existing test-time adaptation (TTA) technology is mainly aimed at the autonomous driving scene, and is difficult to adapt to industrial defect detection. For industrial defect detection, the following technical problems mainly exist: 1. Lack of positive samples: defect samples are scarce, with an average of less than one defect label per image, and no-defect images cannot provide effective pseudo bounding boxes; 2. Low quality of pseudo bounding boxes: the pseudo bounding boxes generated by the model are prone to false positives or position deviations due to domain shift, and the wrong labels will lead to degradation of the model performance; 3. Poor efficiency of Transformer adaptation: although the Transformer architecture can capture long-range dependencies, it has complex parameters and high training difficulty, and is inefficient in target domain adaptation.

[0004] In summary, the application proposes a real-time domain adaptive defect detection method based on double alignment and uncertainty filtering to solve the domain shift problem across scenes and realize high-precision real-time detection of industrial defects in multiple scenes. SUMMARY

[0005] In view of the deficiencies of the prior art, the technical problem to be solved by the present application is to provide a real-time domain adaptive defect detection method based on double alignment and uncertainty filtering.

[0006] The present application solves the technical problem by adopting the following technical scheme: A real-time domain adaptive defect detection method based on double alignment and uncertainty filtering comprises the following steps: Step 1: Construct a defect detection dataset, including source domain data and target domain data; Step 2: Construct a defect detection model; Step 3: Pre-train the defect detection model using the source domain data to obtain a teacher model, and store the multi-scale source domain features extracted by the teacher model backbone network into a feature database according to the scales; initialize the defect detection model using the teacher model parameters to obtain a student model; For a current time step of a current batch, use the teacher model to perform defect detection on a target domain image to generate a pseudo bounding box; calculate the classification entropy and position entropy of the pseudo bounding box and normalize them; if the normalized classification entropy is less than a classification entropy threshold, the normalized position entropy is less than a position entropy threshold, and the pseudo bounding box confidence is greater than a confidence threshold, then the pseudo bounding box is retained, otherwise the pseudo bounding box is removed; use the retained pseudo bounding box as a pseudo label to supervise the training of the student model, and realize the alignment of the pseudo bounding box generated by the teacher model and the bounding box generated by the student model; sum the classification loss and the regression loss of all the bounding boxes to obtain a bounding box alignment loss; For the multi-scale target domain features extracted by the teacher model backbone network, retrieve a plurality of source domain features most similar to the corresponding scale source domain feature group in the feature database, and realize the alignment of the target domain and source domain feature distributions through KL divergence minimization; weight and sum the KL divergence of the target domain features and the source domain features of each scale to obtain a feature alignment loss; Sum the bounding box alignment loss and the feature alignment loss to obtain a total loss; update the parameters of the student model according to the total loss of the current time step, and use the updated student model to detect the target domain image of the next time step, and repeat the above process to update the student model parameters by time step until the detection of the current batch of target domain images is completed; After the detection of the current batch is completed, update the teacher model according to the student model parameters, and repeat the above process for the detection of the next batch; repeat the above process to update the teacher model parameters by batch until the detection of all batches is completed.

[0007] Further, the defect detection model comprises a backbone network, a Transformer encoder, a Transformer decoder and a detection head; the backbone network extracts multi-scale features through a plurality of sparse expansion visual cognitive adaptation modules; the output features of a previous sparse expansion visual cognitive adaptation module are taken as the input features of a next sparse expansion visual cognitive adaptation module after being down-sampled; the sparse expansion visual cognitive adaptation module is based on a shift window Transformer module, and a sparse expansion module is embedded after the two residual connections of the shift window Transformer module.

[0008] Further, in the sparse expansion module, the input features are subjected to layer normalization and then are subjected to residual connection with themselves to obtain first residual connection features; the first residual connection features are subjected to down-sampling and then are split into three sub-features in the channel dimension; the three sub-features are subjected to three parallel dilated convolutions to obtain three dilated convolution features of different scales; the three dilated convolution features are averaged and then are subjected to residual connection with the features obtained by down-sampling to obtain second residual connection features; the second residual connection features are subjected to dot product convolution and then are subjected to residual connection with themselves to obtain third residual connection features; the third residual connection features are subjected to an activation function and up-sampling and then are subjected to residual connection with the input features of the sparse expansion module to obtain the output features of the sparse expansion module.

[0009] Further, the dilation rate of the first dilated convolution is 0, and the receptive field size is 3x3; the dilation rate of the second dilated convolution is 1, and the receptive field size is 5x5; and the dilation rate of the third dilated convolution is 2, and the receptive field size is 7x7.

[0010] Compared with the prior art, the present application has the following beneficial effects: 1. The teacher model and the student model are collaboratively optimized and trained by adopting the pseudo bounding box and feature distribution double alignment strategy, which can not only transfer the source domain defect knowledge through the teacher-student network, but also align the target domain features with the pre-stored source domain features, thereby completely covering the full scene of industrial defect detection and effectively solving the problem that the existing method cannot adapt to the lack of defect samples, thereby alleviating the performance degradation caused by domain shift.

[0011] 2. The uncertainty of the pseudo bounding box is perceived and filtered through the triple standards of position entropy, classification entropy and confidence, the position entropy is used to quantify the positioning uncertainty, and the classification entropy is used to quantify the category uncertainty, so that high-quality pseudo bounding boxes can be accurately selected, the accumulation of false labels is avoided, the usability of the pseudo bounding boxes is significantly improved, the performance stability of the model when adapting to the online domain is ensured, and the problem of low quality of the pseudo bounding boxes caused by the existing method relying only on the confidence threshold or the IoU threshold for filtering is solved.

[0012] 3. To address the issues of easy forgetting of source domain knowledge during full-parameter fine-tuning and redundant structural parameters in real-time domain adaptation of Transformer models, this invention proposes a sparse dilation module embedded in the shift-window Transformer module. The sparse dilation module employs a design combining incremental tuning and sparse dilated convolution. During loss backpropagation, most parameters of the backbone network are frozen, and only the parameters of the sparse dilation module are updated. Simultaneously, dilated convolution maintains receptive fields at different scales, avoiding catastrophic forgetting and significantly reducing computational costs, thus improving domain adaptation efficiency and meeting the needs of real-time factory inspection pipeline operations. This solves the problem of low domain adaptation efficiency inherent in the Transformer architecture. Model updates are synchronized with online detection, eliminating the need for offline processing of target domain data and achieving online dynamic adaptation.

[0013] 4. Pseudo-boundary box alignment does not require target domain data annotation, and feature distribution alignment is adapted through pre-stored source domain features. It does not rely on target domain data annotation throughout the process, achieving adaptation of unannotated target domain data. Attached Figure Description

[0014] Figure 1 This is a structural diagram of the defect detection model of the present invention; Figure 2 This is a structural diagram of the sparse extended visual cognition adaptation module of the present invention; Figure 3 This is a structural diagram of the sparse expansion module of the present invention. Detailed Implementation

[0015] Specific embodiments are given below with reference to the accompanying drawings. These specific embodiments are only used to describe the technical solution of the present invention in detail, and are not intended to limit the scope of protection of this application.

[0016] This invention provides a real-time domain adaptive defect detection method based on double alignment and uncertainty filtering, comprising the following steps: Step 1: Construct a defect detection dataset, including source domain data and target domain data.

[0017] Step 2: Construct a defect detection model based on the RT-DETR end-to-end target detection architecture; such as... Figure 1 As shown, the defect detection model includes a backbone network, a Transformer encoder, a Transformer decoder, and a detection head. The backbone network is used to extract multi-scale features and concatenate the multi-scale features to obtain multi-scale fused features. The multi-scale fused features are encoded by the Transformer encoder to enhance the features of the defect area. The output features of the Transformer encoder are decoded by the Transformer decoder, and the output features of the Transformer decoder are classified and located by the detection head.

[0018] As shown in Figure 1 , the backbone network comprises an embedding layer and a sparse expansion visual cognitive adaptation module; the embedding layer embeds the input image into a feature space through convolution and down-sampling operation to generate initial features; the initial feature map extracts multi-scale features through multiple sparse expansion visual cognitive adaptation modules; the output features of the last sparse expansion visual cognitive adaptation module are down-sampled and used as the input features of the next sparse expansion visual cognitive adaptation module; the sparse expansion visual cognitive adaptation module improves the adaptation efficiency of the backbone network. The backbone network of the embodiment comprises four sparse expansion visual cognitive adaptation modules, which output four scales of features in total.

[0019] As shown in Figure 2 , the sparse expansion visual cognitive adaptation module is based on a shift window Transformer module (SwinTransformer Block), and a sparse expansion module is embedded after the two residual connections of the shift window Transformer module; only the parameters of the sparse expansion module are updated during target domain adaptation, thereby improving the domain adaptation efficiency. As shown in Figure 3 , in the sparse expansion module, the input features are subjected to layer normalization and then subjected to residual connection with themselves to obtain first residual connection features; the first residual connection features are down-sampled and then split into three sub-features in the channel dimension; the three sub-features are subjected to three parallel dilated convolutions to capture different scale features under different receptive fields to obtain three dilated convolution features; the three dilated convolution features are averaged and then subjected to residual connection with the features obtained by down-sampling to obtain second residual connection features; the second residual connection features are subjected to point multiplication convolution to integrate the channel features and then subjected to residual connection with themselves to obtain third residual connection features; the third residual connection features are subjected to an activation function and up-sampling and then subjected to residual connection with the input features of the sparse expansion module to obtain the output features of the sparse expansion module. The sparse expansion module adopts an incremental tuning idea, which can reduce the training parameter amount and improve the model adaptation efficiency while avoiding catastrophic forgetting. The dilation rate of the first dilated convolution is 0, and the receptive field size is 3×3; the dilation rate of the second dilated convolution is 1, and the receptive field size is 5×5; the dilation rate of the third dilated convolution is 2, and the receptive field size is 7×7.

[0020] The Transformer encoder comprises multiple encoding layers, and the embodiment comprises 6 encoding layers; the encoding layers assign weights to each feature through multi-head attention, and the feature weight of the defect area is large and the feature weight of the background area is small, so that the features of “fine scratches” and “large cracks” are strengthened in the same feature to solve the missed detection problem of different scale defects under a single detection scale.

[0021] The Transformer decoder includes multiple decoding layers, six in this embodiment. The decoding layers use deformable cross-attention matching to match features with the target query, sample feature points only in the region corresponding to each candidate box, apply attention weighting to the feature points, generate the defect confidence and position offset of each candidate box, skip the background region, and focus only on the candidate region that may have defects, which reduces the amount of computation and improves the defect localization accuracy.

[0022] The detection head includes a classification branch and a localization branch. The classification branch is used to predict the defect category, and the localization branch is used to predict the defect location to obtain the bounding box.

[0023] Step 3: Using the defect detection dataset, train the defect detection model based on knowledge distillation. Collaborately optimize the teacher model and student model through a pseudo-boundary box and feature distribution dual alignment strategy to achieve model transfer from the source domain to the target domain.

[0024] First, using real labels and source domain data, the defect detection model is pre-trained in a supervised manner. The pre-trained defect detection model is then used as the teacher model. At the same time, the multi-scale source domain features extracted by the backbone network of the teacher model are grouped by scale and stored in the feature database. The defect detection model is initialized using the teacher model parameters to obtain the student model; for the current time step of the current batch, the teacher model is used to perform defect detection on the target domain image to generate pseudo bounding boxes; the uncertainty of the pseudo bounding boxes is perceived through the triple standard of "classification entropy + location entropy + confidence" to filter out low-quality pseudo bounding boxes. Calculate the classification entropy of the pseudo-boundary box to measure the uncertainty of the category judgment; (1) in, It is the classification entropy of the pseudo-boundary box. It is a category The probability distribution, It is the number of categories; Calculate the position entropy of the pseudo-boundary box to measure the uncertainty of the bounding box position; (2) in, It is the position entropy of the pseudo-boundary box. It is the first The boundary of a pseudo-boundary box The offset, , , and These are the left, top, right, and bottom boundaries of the pseudo-boundary box. It represents the number of pseudo-boundary boxes; Normalize the classification entropy and position entropy of the pseudo-boundary box to obtain normalized classification entropy and position entropy; (3) in, It is the normalized entropy , It is entropy The maximum value; If the normalized classification entropy is less than the classification entropy threshold, the normalized location entropy is less than the location entropy threshold, and the confidence of the pseudo-boundary box is greater than the confidence threshold, then the pseudo-boundary box is retained; otherwise, the pseudo-boundary box is discarded. In this embodiment, the classification entropy threshold and the location entropy threshold are both 0.5, and the confidence threshold is 0.25. The student model is used to detect defects in the target domain image to obtain the defect category and location. The retained pseudo-boundary boxes are used as pseudo-labels to supervise the fine-tuning of the student model parameters, so as to achieve the alignment of the pseudo-boundary boxes generated by the teacher model with the bounding boxes generated by the student model. The bounding box alignment loss is calculated by the following formula. (4) in, It is the bounding box alignment loss. It is the first The true category of the bounding box With prediction category Classification loss between It is the first The true location of the bounding box With predicted location The regression loss between them , These are weighting coefficients. It represents the number of bounding boxes; For the multi-scale target domain features extracted by the backbone network of the teacher model, the most similar features are retrieved from the source domain feature groups of the corresponding scale in the feature database. For each source domain feature, the distribution of the target domain features and the source domain features is aligned by minimizing the KL divergence, and the feature alignment loss is calculated by the following formula; (5) in, It is feature alignment loss. It is the first Weights of scale features It is the first The KL divergence between target domain features and source domain features at various scales. These are the target domain features extracted from the backbone network of the teacher model. It is a feature database Features of the target domain Most similar a source domain feature, is a number of dimensions of the feature; The total loss of the current time step is calculated according to the following formula : (6) wherein, is a balance coefficient; The parameters of the student model are updated in reverse propagation according to the total loss of the current time step (only the parameters of the sparse expansion module are updated, and the remaining parameters are frozen), the target domain image of the next time step is detected by using the updated student model, the total loss of the next time step is calculated and the parameters of the student model are updated, and the target domain image is detected by using the updated student model. In this way, the cycle is repeated until the detection of the target domain image of the current batch is completed. After the detection of the current batch is completed, the teacher model is updated according to the parameters of the student model by using the sliding exponential average method, and the above process is repeated for the detection of the next batch. In this way, the cycle is repeated until the detection of all batches is completed. During the domain adaptation process, the parameters of the student model are updated by time step, and the parameters of the teacher model are updated by batch.

[0025] Embodiment: This embodiment is based on the AeBAD data set for expansion, and the CD-AeBD data set is constructed. The HD-AeBD data set is a self-made data set.

[0026] The CD-AeBD data set contains two subsets. The first subset (denoted as CD-AeBD I) contains 1639 images (3024x3024 pixels), including source domain, background change domain, illumination change domain and view angle change domain, a total of 2371 defect frames, and defect types include ablation, fracture, crack and groove. The second subset (denoted as CD-AeBD II) contains 2703 images (800x800 pixels), which are divided into source domain, first view angle and illumination joint change domain and second view angle and illumination joint change domain, and a total of 4050 defect frames.

[0027] The HD-AeBD data set contains 1421 images (1224x2048 pixels), which are divided into B1-B3 domains, and the defect types include recess, corrosion and scratch.

[0028] The test experiment is divided into three groups; the source domain of the first group is the source domain of the first subset of the CD-AeBD dataset (689), the target domain is the background change domain, the illumination change domain and the perspective change domain of the first subset of the CD-AeBD dataset; the source domain of the second group is the illumination change domain of the second subset of the CD-AeBD dataset, and the target domain is the perspective change domain and the illumination and perspective joint change domain of the second subset of the CD-AeBD dataset; the source domain of the third group is the B1 domain (609) of the HD-AeBD dataset, and the target domain is the B2 and B3 domains of the HD-AeBD dataset.

[0029] Select the existing method (including Yolov11, RT-DETR, Define, Co-DETR, T-S, IoU-Filter, ActMAD, CTTAOD) and the method of the application for comparison, and the experimental results on different data sets are shown in Tables 1-3.

[0030] Table 1 Test results (mAP@50) of different methods on CD-AeBD I dataset

[0031] Table 2 Test results (mAP@50) of different methods on CD-AeBD II dataset

[0032] Table 3 Test results (mAP@50 / mAP@50:95) of different methods on HD-AeBD dataset

[0033] mAP@50 represents the average precision when the intersection over union is 0.5, mAP@50:95 represents the average precision when the intersection over union is from 0.5 to 0.95 (with a step of 0.05), Yolov11, RT-DETR, Define, Co-DETR are relatively advanced high-precision target detection models without domain adaptation capability. T-S, IoU-Filter, ActMAD, CTTAOD are real-time domain adaptive detection models, which are updated in real time during testing to adapt to the new target domain. All detection models are trained in the source domain and tested in the target domain. From the above results, although the high-precision target detection model has good performance in the source domain, it cannot effectively handle the domain shift, and the performance in the target domain is significantly reduced. Compared with the real-time domain adaptive detection model, the performance of the present application is significantly better than that of the real-time domain adaptive detection model, because the pseudo bounding box and feature distribution double alignment strategy can simultaneously compare the global feature distribution and instance-level prediction results of the target domain with the source domain, and the model is optimized in real time; at the same time, the pseudo bounding box uncertainty filtering can estimate the uncertainty of the defects in the target domain, so as to more accurately eliminate unreliable samples and avoid the accumulation and amplification of false labels in training.

[0034] In order to verify the effectiveness of the pseudo bounding box and feature distribution double alignment strategy and the pseudo bounding box uncertainty filtering, an ablation experiment is performed on the present application, and the experimental results are shown in Tables 4 and 5.

[0035] Table 4: Test results of different alignment strategies (mAP@50 down)

[0036] Table 5: Test results of different pseudo bounding box filtering standards (mAP@50 down)

[0037] The data in Tables 4 and 5 represent the performance decline of the model during domain adaptation, so the smaller the value, the less the performance decline, indicating better adaptation effect. From Table 4, when no alignment strategy is used, the domain adaptation effect of the model is the worst; when only one alignment strategy is used, the domain adaptation effect is not ideal; when the pseudo bounding box and feature distribution double alignment strategy is used, the domain adaptation performance of the model is the least, which verifies the effectiveness of the pseudo bounding box and feature distribution double alignment strategy. From Table 5, when the confidence, position entropy and classification entropy are used to filter the pseudo bounding box, the false positive targets can be effectively reduced, and a large number of low-quality pseudo bounding boxes can be avoided to accumulate errors, so that the correct bounding box can be better aligned, and therefore the domain adaptation performance of the model is the least, which verifies the effectiveness of the pseudo bounding box uncertainty filtering based on the three standards of confidence, position entropy and classification entropy.

[0038] The invention is applicable where not mentioned.

Claims

1. A real-time domain adaptive defect detection method based on double alignment and uncertainty filtering, characterized in that, Includes the following steps: Step 1: Construct a defect detection dataset, including source domain data and target domain data; Step 2: Construct a defect detection model; Step 3: Use source domain data to pre-train the defect detection model to obtain the teacher model. At the same time, group the multi-scale source domain features extracted by the backbone network of the teacher model and store them in the feature database according to scale. The defect detection model is initialized using the teacher model parameters to obtain the student model; For the current time step of the current batch, the teacher model is used to perform defect detection on the target domain image to generate pseudo bounding boxes; the classification entropy and position entropy of the pseudo bounding boxes are calculated and normalized. If the normalized classification entropy is less than the classification entropy threshold, the normalized positional entropy is less than the positional entropy threshold, and the false bounding box confidence is greater than the confidence threshold, then the false bounding box is retained; otherwise, the false bounding box is discarded. The retained false bounding boxes are used as pseudo-labels to supervise the training of the student model, thereby aligning the false bounding boxes generated by the teacher model with the bounding boxes generated by the student model. The classification loss and regression loss of all bounding boxes are summed to obtain the bounding box alignment loss. For the multi-scale target domain features extracted by the backbone network of the teacher model, the most similar source domain features are retrieved from the source domain feature groups of the corresponding scale in the feature database. The alignment of the target domain and source domain feature distributions is achieved by minimizing the KL divergence. The KL divergence of the target domain features and source domain features at each scale is weighted and summed to obtain the feature alignment loss. The bounding box alignment loss and feature alignment loss are summed to obtain the total loss. The student model parameters are updated by backpropagation based on the total loss at the current time step. The updated student model is then used to detect the target domain image at the next time step. This process is repeated, updating the student model parameters step by step, until the detection of the current batch of target domain images is completed. After completing the current batch of testing, update the teacher model using the student model parameters, and repeat the above process for the next batch of testing; this cycle continues, updating the teacher model parameters batch by batch, until all batches of testing are completed.

2. The real-time domain adaptive defect detection method based on double alignment and uncertainty filtering according to claim 1, characterized in that, The defect detection model includes a backbone network, a Transformer encoder, a Transformer decoder, and a detection head. The backbone network extracts multi-scale features through multiple sparse extended visual cognition adaptation modules. The output features of the previous sparse extended visual cognition adaptation module are downsampled and used as the input features of the next sparse extended visual cognition adaptation module. The sparse expansion visual cognition adaptation module is based on the shift window Transformer module, and a sparse expansion module is embedded after the two residual connections of the shift window Transformer module.

3. The real-time domain adaptive defect detection method based on double alignment and uncertainty filtering according to claim 2, characterized in that, In the sparse dilation module, the input features are normalized by layers and then residually concatenated with themselves to obtain the first residual concatenated feature. The first residual concatenated feature is downsampled and then split into three sub-features along the channel dimension. The three sub-features undergo three parallel dilated convolutions to obtain three dilated convolution features at different scales. The three dilated convolution features are averaged and then residually concatenated with the downsampled features to obtain the second residual concatenated feature. The second residual concatenated feature undergoes dot-product convolution and then residually concatenates with itself to obtain the third residual concatenated feature. The third residual concatenated feature undergoes activation function and upsampling, and then residually concatenates with the input features of the sparse dilation module to obtain the output features of the sparse dilation module.

4. The real-time domain adaptive defect detection method based on double alignment and uncertainty filtering according to claim 3, characterized in that, The first dilated convolution has a dilation rate of 0 and a receptive field size of 3×3; the second dilated convolution has a dilation rate of 1 and a receptive field size of 5×5; and the third dilated convolution has a dilation rate of 2 and a receptive field size of 7×7.

Images