Fuzzy labeling defect detection method based on brightness adaptation enhancement and content awareness
By adopting fuzzy label defect detection methods with brightness adaptation enhancement and content perception in industrial defect detection, the data label accuracy and consistency problems caused by fuzzy labeling are solved, and the detection accuracy and recognition capabilities of the model are improved.
Patent Information
- Application Number
- CN202411670630.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-21
- Publication Date
- 2025-02-28
AI Technical Summary
The prior art faces the accuracy and consistency of data labeling caused by fuzzy labeling in industrial defect detection, which affects the detection performance of the model.
The fuzzy labeling defect detection method based on brightness adaptation enhancement and content perception is adopted. Through technical means such as brightness adaptation data enhancement and deformable attention module, the accuracy and consistency of data labeling are improved and the model's ability to identify defect features is enhanced.
It improves the detection accuracy of the model, reduces the false detection and missed detection rates, and enhances the model's ability to identify defects with blurred boundaries or difficult to judge.
Smart Images
Figure CN119540955A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of industrial defect detection, and specifically is a fuzzy annotation defect detection method based on brightness adaptation enhancement and content perception. Background Art
[0002] In complex production processes and changing production environments, cracks, scratches, spots and other defects often appear on the surface of industrial products. These defects not only affect the appearance quality of the product, but also may reduce its corrosion resistance and mechanical strength, increase its risk of breakage during use, and ultimately threaten the safety and reliability of the product. Vision-based defect detection technology came into being, and with its high-precision and efficient detection capabilities, it is widely used in the quality control of industrial products.
[0003] In recent years, target detection methods based on deep learning have been used for defect detection, such as Faster R-CNN, SSD, and YOLO series. These methods show strong real-time and detection performance when processing large-scale data, significantly improving the automation level and detection efficiency of industrial production lines. However, in practical applications, they still face the problem of accuracy and consistency of data annotation. Industrial defect detection usually requires a large amount of high-quality annotated data for model training and verification. However, due to the diversity of defect types, the inconsistency of annotation standards, and the deviation of annotation personnel's understanding of the severity of defects, the annotation results are ambiguous, such as the difference between the same target when multiple people annotate, and the difference between the same target before and after a single person annotates, collectively referred to as fuzzy annotation. Fuzzy annotation mainly affects model performance at two levels: on the one hand, fuzzy annotation will cause the model to be unable to accurately distinguish defects of similar categories during training, thereby affecting the convergence and prediction effect of the model; on the other hand, data with fuzzy labels may increase the false detection rate and missed detection rate of the model, especially when facing defects with blurred boundaries or difficult to judge, the performance of the model is often unstable. Based on this, the present application proposes a fuzzy annotation defect detection method based on brightness adaptation enhancement and content perception. Summary of the invention
[0004] In view of the deficiencies in the prior art, the technical problem to be solved by the present invention is to provide a fuzzy annotation defect detection method based on brightness adaptation enhancement and content perception.
[0005] The present invention solves the technical problem by adopting the following technical solution: A fuzzy annotation defect detection method based on brightness adaptation enhancement and content perception, characterized in that the method comprises the following steps: Step 1: Obtain a fuzzy labeled dataset and perform data enhancement; First, the images of defects of the same category are converted into grayscale images, and the sum of all pixel values in the grayscale images is calculated to obtain the average brightness of the image; the absolute value of the difference in average brightness between any two images is calculated. If the absolute value is less than or equal to the brightness difference threshold, the two images are allowed to enhance each other, otherwise, mutual enhancement is not allowed. Then, the images of defects of the same category are screened to select the image group that allows mutual enhancement; Then, according to the defect annotation information, the probability density map of the defect is generated using the KDE algorithm, and the common location of the defect is determined based on the probability density map; one image in the image group that allows mutual enhancement is used as the source image, and the other image is used as the target image. According to the common location of the defect, multiple candidate locations are generated on the target image using the NMS algorithm, and the candidate location with the highest confidence is selected as the pasting location; all pixels of a random defect in the source image are copied and pasted to the pasting location of the target image to obtain an enhanced image; all image groups that allow mutual enhancement are traversed to complete data enhancement, and then the enhanced data set is obtained; Step 2: Build a defect detection model; the defect detection model includes a feature extraction network based on deformable attention, a content-aware efficient aggregation feature fusion network and a detection head, and the feature extraction network based on deformable attention includes a backbone network of the YOLOv8 model and a deformable attention module; The input image is extracted through the backbone network of the YOLOv8 model. The output features of the last three feature extraction layers of the backbone network of the YOLOv8 model are respectively passed through the deformable attention module to obtain the output features of the feature extraction network based on the deformable attention. , and ; The feature , and Input into the content-aware efficient aggregation feature fusion network, the feature Features and features obtained after downsampling Through a feature fusion block, the feature ;feature Features and features obtained by CARAFE upsampling And features The downsampled features are fused through a feature fusion block to obtain the feature ;feature Features and features obtained by CARAFE upsampling Through a feature fusion block, the feature ;feature After convolution, we get the features ;feature and The features and features obtained by downsampling respectively Through a feature fusion block, the feature ;feature and The features and features obtained by downsampling respectively Through a feature fusion block, the feature ; The content-aware efficient aggregation features are fused with the output features of the network , as well as Input into the detection head for detection and obtain the detection result; Step 3: Use the enhanced data set to train the defect detection model, and use the trained defect detection model for defect detection.
[0006] Furthermore, during the model training process, the training loss is calculated using the loss function of formula (2); (2) In the formula, is the intersection-over-union ratio between the predicted box and the true box, is the Euclidean distance between the center points of the predicted box and the true box, is the diagonal length of the minimum enclosing rectangle containing the predicted box and the true box, is the equilibrium parameter, is a measure of aspect ratio consistency, It is the KL divergence between the predicted box and the true box.
[0007] Compared with the prior art, the present invention has the following beneficial effects: 1. At the data enhancement level, in response to the situation where industrial defects have fuzzy annotations, the present invention proposes a brightness adaptation data enhancement method, which follows the key idea of copy and paste data enhancement. However, unlike copy and paste data enhancement, brightness adaptation data enhancement fully considers the brightness difference between the source image and the target image. Only when the brightness difference meets the conditions, the two images are allowed to enhance each other, avoiding large brightness differences between defects and backgrounds in the enhanced image, thereby avoiding the model from mistakenly identifying difference information as defect information, which is beneficial to improving the false detection rate and missed detection rate of the model. In the enhancement process, the annotation information of the defect is fully utilized to determine the pasting position, and the pasting position is selected at the common position of the defect, thereby improving the visual consistency between the defect and the target image, and avoiding the defect from appearing abrupt in the target image.
[0008] 2. At the network model level, for the fuzzy labeling problem, since the defects will be completely labeled no matter how they are labeled, the model uses a deformable attention module and a content-aware efficient aggregation feature fusion network to guide at the feature level, so that the model can focus more on the common characteristics of the defects, which is conducive to improving detection accuracy and further avoiding missed detections and false detections.
[0009] 3. When constructing the loss function, the bounding box is modeled as divergence information to improve the constraint on the bounding box, and the KL divergence loss is combined with the CIoU loss to improve the detection confidence while ensuring the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 Schematic diagram of data augmentation for brightness adaptation; Figure 2 It is the structural diagram of the defect detection model; Figure 3 Different annotation results for the same defect. DETAILED DESCRIPTION
[0011] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods, but the protection scope of the present application is not limited thereto.
[0012] The present invention provides a fuzzy annotation defect detection method based on brightness adaptation enhancement and content perception (hereinafter referred to as method, see Figures 1 to 3 ), including the following steps: Step 1: Obtain a fuzzy labeled data set, perform data enhancement using brightness adaptation data enhancement, and obtain an enhanced data set; The key idea of copy-paste data enhancement is to copy and paste the object from the source image to the target image. For industrial inspection tasks, since the background is similar, no additional pixel annotation is required. Therefore, it is feasible to directly copy and paste all pixels in the source image bounding box to the target image. However, when there is a significant difference in background brightness or texture between the source image and the target image, the model may mistakenly identify the background difference as a defect, thereby affecting the detection accuracy. Brightness adaptation data enhancement follows the key idea of copy-paste data enhancement, fully considers the brightness similarity between images, and determines whether the two images are allowed to be enhanced mutually by measuring the brightness difference between the two images, thereby screening out the image group that is allowed to be enhanced mutually. In the enhancement process, the defect annotation information is fully utilized to determine the pasting position, and the pasting position is selected at the common position of the defect, thereby improving the visual consistency between the defect and the target image, and avoiding the defect appearing abrupt in the target image.
[0013] First, all images of defects of the same category are screened to select the image group that allows mutual enhancement. The image is converted into a grayscale image, and all pixel values of the image are summed up by formula (1) to obtain the average brightness of the image. The absolute value of the difference in average brightness between any two images is calculated. If the absolute value is less than or equal to the brightness difference threshold, the two images are allowed to be enhanced mutually. Otherwise, mutual enhancement is not allowed. Thus, the image group that allows mutual enhancement is obtained. (1) In the formula, is the average brightness of the image, , are the height and width of the image, Indicates location The pixel value at ; Then, according to the defect annotation information (such as bounding box, defect category, etc.), the kernel density estimation (KDE) algorithm is used to generate a probability density map of the defect, and the common location of the defect is determined based on the probability density map; one image in the image group that allows mutual enhancement is used as the source image, and the other image is used as the target image. According to the common location of the defect, the NMS algorithm is used to generate multiple candidate locations on the target image, and the candidate location with the highest confidence is selected as the pasting location; all pixels of a random defect in the source image are copied and pasted to the pasting location of the target image to obtain an enhanced image; all image groups that allow mutual enhancement are traversed to complete data enhancement, and then the enhanced data set is obtained.
[0014] Step 2: Build a defect detection model (DACAR-Net); Figure 2 As shown in the figure, the defect detection model includes a feature extraction network based on deformable attention, a content-aware efficient aggregation feature fusion network and a detection head; the input image is subjected to feature extraction by the feature extraction network based on deformable attention to obtain the feature , and ;feature , and The feature fusion is performed through the content-aware efficient aggregation feature fusion network to obtain the feature , as well as ;feature , as well as The detection is carried out by the detection head to obtain the detection result.
[0015] The feature extraction network based on deformable attention includes a backbone network of the YOLOv8 model and a deformable attention module. The input image is subjected to layer-by-layer feature extraction through the backbone network of the YOLOv8 model. The output features c3, c4, and c5 of the last three feature extraction layers of the backbone network of the YOLOv8 model are respectively subjected to the deformable attention module to obtain the feature , and , which is the output feature of the feature extraction network based on deformable attention. The deformable attention module can dynamically adjust the shape and size of the attention model by using the deformable attention mechanism. In feature extraction, the channel and spatial feature information are enhanced by introducing the deformable attention module. Under the guidance of the feature important area, the deformable attention module effectively models the relationship between labels, so that the model pays more attention to the information representing the defects.
[0016] In the content-aware efficient aggregation feature fusion network, the features Features and features obtained after downsampling Through a feature fusion block, the feature ;feature Features and features obtained by CARAFE upsampling And features The downsampled features are fused through a feature fusion block to obtain the feature ;feature Features and features obtained by CARAFE upsampling Through a feature fusion block, the feature ;feature After convolution, we get the features ;feature and The features and features obtained by downsampling respectively Through a feature fusion block, the feature ;feature and The features and features obtained by downsampling respectively Through a feature fusion block, the feature ;feature , as well as That is, the output feature of the content-aware efficient aggregation feature fusion network.
[0017] The content-aware efficient aggregation feature fusion network uses CARAFE upsampling to enable features to more accurately represent defect shapes and enhance the recognition ability of feature mapping, so that the model can better predict the detection results; at the same time, it realizes accurate information transmission between features at different levels, which is conducive to improving detection accuracy. The feature fusion block uses a cross-stage local network (CSPNet) to replace the traditional feature fusion method based on 3×3 convolution, achieving richer gradient combinations while reducing the amount of calculation.
[0018] Step 3: Use the enhanced data set to train the defect detection model, calculate the training loss through the loss function of formula (2), and use the trained defect detection model for defect detection; (2) In the formula, is the intersection-over-union ratio between the predicted box and the true box, is the Euclidean distance between the center points of the predicted box and the true box, is the diagonal length of the minimum enclosing rectangle containing the predicted box and the true box, is the equilibrium parameter, is a measure of aspect ratio consistency, It is the KL divergence between the predicted box and the true box.
[0019] As can be seen from formula (2), the loss function combines the KL divergence and CIoU loss function. The CIoU loss function provides rich gradient information by comprehensively considering IoU, center point distance and aspect ratio, so that the model can converge faster during training. The KL divergence enhances the constraints of the model on prediction boxes at different positions and improves the positioning confidence of the model.
[0020] Example: This example is based on the Northeastern University strip steel defect classification dataset (denoted as NEU-1, which simulates the fuzzy labeling problem caused by a single person), and uses multiple people to re-label the defects to simulate the scene of multiple people labeling fuzzy, and obtains three datasets with different annotation standards, denoted as NEU-2, NEU-3 and NEU-4. The different labeling results of the same defect are shown in Figure 3 Therefore, this embodiment includes four data sets in total, each of which includes images of three types of defects: patches, inclusions, and scratches, and each type of defect includes 300 images.
[0021] In order to verify that the proposed method can improve the defect detection accuracy under different labeling standards, the NEU-1~NEU-4 datasets are used as training sets, and the remaining three datasets are tested and compared with the baseline model (Yolov8). The detection accuracy is shown in Table 1.
[0022]
[0023] As shown in Table 1, taking the second row as an example, NEU-1 is the training set, NEU-2, NEU-3 and NEU-4 are the test sets, and the detection accuracy on the same data set reaches 92.8%, while the accuracy of the other three data sets is between 64.8% and 84.5%, indicating that the detection accuracy of the model will be very different when the annotation standards are different. When the training set and the test set are the same, the detection accuracy of the method of the present invention is improved compared with the baseline model, indicating that the method of the present invention effectively alleviates the fuzzy annotation problem and can accurately detect and locate defects in the case of fuzzy annotation.
[0024] In order to verify that the method of the present invention has a better detection effect, the method of the present invention and the common target detection model are experimented on the NEU-1 dataset. The detection results of different models are shown in Table 2. Table 3 shows the ablation experiment results of the method of the present invention on the NEU-1 dataset.
[0025]
[0026]
[0027] As can be seen from Table 2, compared with the common target detection model, the method of the present invention shows good detection effects on three defects: patches, inclusions and scratches, and the accuracy rate is improved to a certain extent. In Table 3, DACAR represents the content-aware efficient aggregation feature fusion network, and KLIoU represents the loss function of the present invention. As can be seen from Table 3, the defect detection model of the present invention has good detection effects on different data sets.
[0028] The above experimental results show that under the fuzzy labeling conditions of multiple people, the accuracy of the defect detection model of the present invention is greatly improved, and the accuracy difference between data sets is narrowed; under the fuzzy labeling conditions of inconsistent labeling before and after a single person, the defect detection model of the present invention also shows superiority. Therefore, the method of the present invention has a good detection effect. There are three main reasons. First, through brightness adaptation data enhancement, defects and annotations are made more diverse, enriching the visual representation of training samples, so that the model can fully learn more defect features during the training process, so that it can accurately distinguish defects of similar categories, and perform more stably when facing defects with blurred boundaries or difficult to judge. Second, through the deformable attention module and the content-aware efficient aggregation feature fusion network, guidance is carried out at the feature level to capture the morphological changes of defects and the visual features of diversified annotations, so that the model can focus more on the common features of defects, which is conducive to improving detection accuracy. Third, the bounding box regression loss is modified to the KLIoU loss, and the constraint of the bounding box divergence is added, so that the model can improve the positioning accuracy.
[0029] The sources of the above models are as follows: [1] Ren S, He K, Girshick R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39(6): 1137-1149. [2] Ross T Y, Dollár G. Focal loss for dense object detection[C] / / proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 2980-2988. [3] Yu X, Lyu W, Zhou D, et al. ES-Net: Efficient scale-aware network for tiny defect detection[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-14. [4] Cai L, Zhang Z, Zhu Y, et al. Bigdetection: A large-scale benchmark for improved object detector pre-training[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 4777-4787. [5] Su J, Luo Q, Yang C, et al. PMSA-DyTr: Prior-Modulated andSemantic-Aligned Dynamic Transformer for Strip Steel Defect Detection[J]. IEEE Transactions on Industrial Informatics, 2024. [6] Terven J, Córdova-Esparza DM, Romero-González J A. Acomprehensive review of yolo architectures in computer vision: From yolov1 toyolov8 and yolo-nas[J]. Machine Learning and Knowledge Extraction, 2023, 5(4): 1680-1716. Any matters not described in the present invention are applicable to the prior art.
Claims
1. A fuzzy annotation defect detection method based on brightness adaptation enhancement and content perception, characterized in that: The method comprises the following steps: Step 1: Obtain a fuzzy labeled dataset and perform data enhancement; First, the images of defects of the same category are converted into grayscale images, and the sum of all pixel values in the grayscale images is calculated to obtain the average brightness of the image; the absolute value of the difference in average brightness between any two images is calculated. If the absolute value is less than or equal to the brightness difference threshold, the two images are allowed to enhance each other, otherwise, mutual enhancement is not allowed. Then, the images of defects of the same category are screened to select the image group that allows mutual enhancement; Then, according to the defect annotation information, the probability density map of the defect is generated using the KDE algorithm, and the common location of the defect is determined based on the probability density map; one image in the image group that allows mutual enhancement is used as the source image, and the other image is used as the target image. According to the common location of the defect, multiple candidate locations are generated on the target image using the NMS algorithm, and the candidate location with the highest confidence is selected as the pasting location; all pixels of a random defect in the source image are copied and pasted to the pasting location of the target image to obtain an enhanced image; all image groups that allow mutual enhancement are traversed to complete data enhancement, and then the enhanced data set is obtained; Step 2: Build a defect detection model; the defect detection model includes a feature extraction network based on deformable attention, a content-aware efficient aggregation feature fusion network and a detection head, and the feature extraction network based on deformable attention includes a backbone network of the YOLOv8 model and a deformable attention module; The input image is extracted through the backbone network of the YOLOv8 model. The output features of the last three feature extraction layers of the backbone network of the YOLOv8 model are respectively passed through the deformable attention module to obtain the output features of the feature extraction network based on the deformable attention. , and ; The feature , and Input into the content-aware efficient aggregation feature fusion network, the feature Features and features obtained after downsampling Through a feature fusion block, the feature ;feature Features and features obtained by CARAFE upsampling And features The downsampled features are fused through a feature fusion block to obtain the feature ;feature Features and features obtained by CARAFE upsampling Through a feature fusion block, the feature ;feature After convolution, we get the features ;feature and The features and features obtained by downsampling respectively Through a feature fusion block, the feature ;feature and The features and features obtained by downsampling respectively Through a feature fusion block, the feature ; The content-aware efficient aggregation features are fused with the output features of the network , as well as Input into the detection head for detection and obtain the detection result; Step 3: Use the enhanced data set to train the defect detection model, and use the trained defect detection model for defect detection.
2. The fuzzy annotation defect detection method based on brightness adaptation enhancement and content perception according to claim 1 is characterized in that: During the model training process, the training loss is calculated using the loss function of formula (2); （2） In the formula, is the intersection-over-union ratio between the predicted box and the true box, is the Euclidean distance between the center points of the predicted box and the true box, is the diagonal length of the minimum enclosing rectangle containing the predicted box and the true box, is the equilibrium parameter, is a measure of aspect ratio consistency, It is the KL divergence between the predicted box and the true box.
3. The fuzzy annotation defect detection method based on brightness adaptation enhancement and content perception according to claim 1 or 2, characterized in that: The feature fusion block adopts a cross-stage local network CSPNet.