Small sample steel defect detection method based on attention feature pyramid mechanism

Abstract

This invention discloses a small-sample steel defect detection method based on an attention feature pyramid mechanism. First, an initial multi-scale feature map is obtained via a convolutional neural network. Then, using a top-down and lateral connection approach, an improved multi-scale attention network is used to process each of the initial multi-scale feature maps to obtain intermediate multi-scale feature maps. Next, using a bottom-up and lateral connection approach, the improved multi-scale attention network is used to process each of the intermediate multi-scale feature maps to obtain the final multi-scale feature map. Finally, the final classification result is output. A pseudo-label semi-supervised learning algorithm is used to train the detection network model. A local linear embedding algorithm is used to reduce the dimensionality and visualize the high-level features output from the training. Mislabeled or incorrectly labeled images are removed and replaced with unlabeled image data. The model is then retrained to generate pseudo-labels to expand the training set until the training requirements are met. The image to be detected is then input into the trained detection network model for recognition.

Description

Technical Field

[0001] This invention belongs to the technical field of artificial intelligence and relates to a small-sample steel defect detection method based on the attention feature pyramid mechanism. Background Technology

[0002] The importance of surface quality in steel products, especially cold-rolled steel, has grown since the 1980s, primarily due to the demands of automobile manufacturers. Over time, the surface quality of hot-rolled steel, and indeed all steel products, has become extremely important. However, during steel production, various defects can occur on the steel surface due to environmental or human factors. Therefore, defect detection is a crucial step that must be completed after production.

[0003] Currently, most steel mills rely on quality inspectors to detect steel defects. However, due to eye fatigue, traditional manual visual inspection methods are inefficient and have a high rate of missed detections. Furthermore, inconsistent defect detection standards among different inspectors also affect the effectiveness of steel defect detection. With the development of artificial intelligence, AI can now achieve detection accuracy exceeding that of humans. Machines are not constrained by external conditions such as environment and climate, and they offer longer working hours and lower costs than humans. Therefore, machine vision technology in the field of artificial intelligence is gradually being applied in the industrial sector.

[0004] Traditional machine vision inspection first involves manually extracting features from images. However, steel defects are usually small, and the human eye is prone to errors over long periods of time. Furthermore, feature extraction is challenging, resulting in high manual costs and low efficiency. In addition, in the era of big data, the amount of industrial image data that needs to be inspected is enormous, and traditional machine vision is not suitable for processing such large datasets. These issues make it difficult for traditional machine vision to meet the requirements of industrial applications in terms of accuracy and robustness.

[0005] Deep learning inspection has gradually replaced traditional machine vision inspection. The advantage of deep learning is that it can automatically extract features. Compared with manual feature extraction, deep learning is not only more efficient and less expensive, but also more accurate in understanding sample features. Therefore, the accuracy of deep learning is higher than that of traditional machine vision. However, deep learning inspection also requires a large number of samples to ensure sufficient accuracy. It is difficult to collect enough useful samples on a real steel production line. Therefore, defect detection methods that require a large number of labeled samples cannot be directly applied to the detection of defects on the surface of steel.

[0006] Given the above, there are few samples of steel surface defects and the collection process is difficult. In the real world, there is an urgent need to detect steel surface defects when the sample size is limited. Summary of the Invention

[0007] The purpose of this invention is to overcome the technical shortcomings of traditional deep learning algorithms for steel plate surface defect detection, such as the need for a large number of samples to ensure sufficient accuracy. It provides a small-sample steel defect detection method based on the attention feature pyramid mechanism. Based on the optimization of the fine-tuned small-sample target detection method, the feature pyramid, attention mechanism and semi-supervised learning algorithm modules have been improved, innovated and optimized to improve the accuracy and robustness of steel defect detection.

[0008] This invention can be achieved through the following technical solutions:

[0009] A small-sample steel defect detection method based on attention feature pyramid mechanism includes: 1. Constructing a detection network model.

[0010] First, an initial multi-scale feature map is obtained through a convolutional neural network. Then, an improved multi-scale attention network is used to process the initial multi-scale feature map one by one using a top-down and lateral connection approach to obtain intermediate multi-scale feature maps. Next, an improved multi-scale attention network is used to process the intermediate multi-scale feature maps one by one using a bottom-up and lateral connection approach to obtain the final multi-scale feature map. Finally, the final multi-scale feature maps are fused and sequentially input into the RPN network, ROI pooling layer, classifier, and regressor to output the final classification result.

[0011] II. Training and Detection Network Model

[0012] The detection network model is trained using a pseudo-label semi-supervised learning algorithm. The high-level features of the training output are reduced in dimensionality and visualized using the local linear embedding algorithm. Mislabeled images are removed and replaced with unlabeled image data. The training is then retrained to generate pseudo-labels, thereby expanding the training set until the training requirements are met.

[0013] III. Testing

[0014] The image to be inspected is input into a trained detection network model for recognition.

[0015] Furthermore, the improved multi-scale attention network includes a split module and serial channel attention modules and spatial attention modules.

[0016] The Split module is used to extract features from multi-scale feature maps and aggregate them into an input feature map with a scale of C*H*W;

[0017] The channel attention module is used to divide the input feature map of scale C*H*W into four branches, namely A, B, C, and D from top to bottom. After reshaping branches A and B, a feature map of scale C×N is generated. Then, the result of multiplying the transpose of branch B and the transpose of branch A is subjected to a Softmax operation to generate a feature map of scale C×C. Then, the transpose of the feature map of scale C×C is multiplied with the reshaped C×N branch, and then reshaped back to a feature map of size C*H*W and multiplied by a coefficient β. Finally, it is added to branch D to obtain a channel feature map that incorporates channel information.

[0018] The spatial attention module first uses three convolutional kernels to generate three new feature maps from the channel feature map of size C*H*W, which are the feature maps of branches A, B, and C from top to bottom. Then, branches A and B are reshaped to generate a feature map of size C×N. The result of multiplying the transpose of branches B and A is then subjected to a Softmax operation to generate a feature map of size N×N. After that, the transpose of the N×N feature map is multiplied with the reshaped C branch, and then reshaped back to a feature map of size C*H*W and multiplied by a coefficient α. Finally, it is added to the original channel feature map to obtain a spatial feature map that incorporates spatial information.

[0019] Furthermore, the convolutional neural network employs a ResNet101 residual network.

[0020] Furthermore, firstly, the input image is processed through the optimal backbone network of ResNet101 to obtain feature maps C2, C3, C4, and C5 at different scales, i.e., the initial multi-scale feature maps. Next, feature map C5 is input into an improved multi-scale attention network to obtain a feature map that integrates channel attention weights and spatial attention weights. This feature map is then added to feature map P5 after a 1×1 convolution. Subsequently, feature maps P4, P3, and P2 at different scales are obtained through top-down and lateral connections, i.e., intermediate multi-scale feature maps. Similarly, feature map P2 is processed through the improved multi-scale attention network and added to feature map P2 after a 1×1 convolution to obtain feature map F2. This feature map F2 is then obtained through bottom-up and lateral connections, i.e., the final multi-scale feature maps. Finally, these final multi-scale feature maps are fused and input into the RPN network, the ROIPooling layer, and the two branches of the classifier and regressor to obtain the final output result.

[0021] Furthermore, training the detection network model includes the following steps:

[0022] (1) A pseudo-label semi-supervised learning algorithm is adopted, and a detection network model is trained using labeled small sample image data;

[0023] (2) Extract the high-level features of each training result image, perform dimensionality reduction and visualization processing on the high-level features through the Local Linear Embedding (LLE) algorithm, then perform data cleaning, filter out mislabeled images and remove the labels, and put in unlabeled image data.

[0024] (3) Use the trained detection network model to predict unlabeled image data and use the prediction results as pseudo-labels;

[0025] (4) Combine the labeled small sample image data and pseudo-labels as an expanded training set to continue training the detection network model;

[0026] (5) Repeat steps 2 and 3 until the detection network model converges and achieves the expected effect, or reaches the preset number of training times, to obtain the final detection network model.

[0027] Beneficial effects:

[0028] 1. Construction of a small sample dataset based on semi-supervised learning. To reduce reliance on manually labeled data, a pseudo-label semi-supervised learning method is used, which can significantly solve this problem. Finally, the steel defect dataset is divided into categories according to the small sample standard. The dataset is divided into base class and new class in a 1:1 ratio to construct a small sample dataset.

[0029] 2. Research on Small Sample Target Detection Based on Fine-tuning Method. Due to the high accuracy requirements for steel surface defect detection in real-world applications, this invention selects a fine-tuning-based small sample learning method from common methods. Based on this, experimental comparisons are conducted on basic target detection frameworks to select the framework with the best accuracy, and improvements will be made to this framework.

[0030] 3. A small target detection algorithm based on feature fusion and attention mechanism. First, to address the problem of small steel defects with large spans that are difficult to detect, feature fusion or feature pyramid modules are added to reduce the loss of structural information. Second, to address the issues of false positives and false negatives of small target defects, an attention module is added so that the model can focus on the features of small target defects in the steel, thereby improving the algorithm's accuracy. Finally, the semi-supervised learning algorithm is optimized to assist in model training, further improving the model's accuracy. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0032] Figure 2 This is a schematic diagram of the structure of the improved multi-scale attention network of the present invention;

[0033] Figure 3 An abstract demonstration diagram illustrating the dimensionality reduction process of the LLE algorithm of this invention;

[0034] Figure 4 This is a schematic diagram comparing the detection results of a single defect when using the small sample steel defect detection method of the present invention and existing methods.

[0035] Figure 5 This is a schematic diagram comparing the results of steel defect detection using different detection models;

[0036] Figure 6(a) , 6(b) 6(c) and 6(d) are schematic diagrams of the visualization results of pit defects, crack defects, scratch defects and iron oxide scale defects when the small sample steel defect detection method of the present invention is used for detection.

[0037] Figure 7 This is a schematic diagram showing the coexistence of two types of defects when using the small-sample steel defect detection method of the present invention.

[0038] Figure 8 This is a schematic diagram showing the coexistence of three types of defects when using the small-sample steel defect detection method of the present invention. Detailed Implementation

[0039] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0040] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0041] like Figure 1 As shown, this invention proposes a small-sample steel defect detection method based on an attention feature pyramid mechanism. It optimizes the small-sample target detection method by fine-tuning the feature pyramid, attention mechanism and semi-supervised learning algorithm, and improves, innovates and optimizes modules such as feature pyramid, attention mechanism and semi-supervised learning algorithm to improve the accuracy and robustness of steel defect detection.

[0042] Specifically as follows:

[0043] I. Constructing a Detection Network Model

[0044] First, an initial multi-scale feature map is obtained through a convolutional neural network. Then, an improved multi-scale attention network is used to process the initial multi-scale feature map one by one using a top-down and lateral connection approach to obtain intermediate multi-scale feature maps. Next, an improved multi-scale attention network is used to process the intermediate multi-scale feature maps one by one using a bottom-up and lateral connection approach to obtain the final multi-scale feature map. Finally, the final multi-scale feature maps are fused and sequentially input into the RPN network, ROI pooling layer, classifier, and regressor to output the final classification result.

[0045] Through research and comparative analysis of small-sample target detection methods, this invention found that metric learning-based methods lack sufficient accuracy in target localization; data augmentation methods, which expand small-sample datasets into regular target detection datasets, follow the same experimental steps as regular target detection, failing to meet the current realities and research objectives of steel surface defect detection; and meta-learning-based methods may suffer from limited generalization ability of the meta-model when handling complex tasks, leading to poor performance. Therefore, this invention ultimately determined to adopt a fine-tuning-based small-sample target detection method, which achieves higher detection accuracy and aligns with the current realities and research objectives of steel surface defect detection.

[0046] Few-shot object detection based on fine-tuning was first proposed by Kang et al. It mainly provides few-shot learning capabilities by making full use of the detection training data from the base class and quickly adjusting the detection prediction network to predict new classes based on a few supporting examples.

[0047] The model consists of two stages. The first stage involves pre-training a base model on a large-scale labeled training set, such as the commonly used Faster R-CNN or YOLO. This base model can be trained on large-scale datasets like ImageNet and has good generalizability. The second stage involves fine-tuning on a small training set. Typically, a small training set contains only tens to hundreds of labeled images, and the types and quantity of labeled data are limited. In this stage, the model's parameters need to be adjusted to adapt to the small dataset and improve the model's performance on the object detection task. During fine-tuning, strategies similar to the learning rate decay in the first stage can be used to ensure the model's convergence and robustness.

[0048] This invention introduces the fine-tuning-based few-sample object detection method proposed by Wang et al. They use the widely used two-stage object detector Faster R-CNN as the basic detection model. The Faster R-CNN model includes a backbone network (e.g., ResNet, VGG16), a region proposal network (RPN), and a two-layer fully connected (FC) sub-network as a proposal-level feature extractor. It also includes a classifier for classifying object categories and a regressor for predicting boundary coordinates.

[0049] The first stage is actually the same as other object detection frameworks: training the network with a large number of base class samples, using the following loss function:

[0050] L = L rpn +L cls +L loc

[0051] Among them, L rpn The output of RPN is used to distinguish foreground and background and refine anchor points, L cls It is the cross-entropy loss of the box classifier C, L loc It is the smoothed L1 loss of the box regressor R.

[0052] The second stage is the fine-tuning stage with a small number of samples. The dataset used includes a small number of base classes and new classes. First, the network is assigned randomly initialized weights to the new classes. While keeping the entire feature extractor fixed, only the bounding box classification and regression networks are fine-tuned.

[0053] In the second stage, a cosine similarity-based classifier is used because, in the case of small samples, if the traditional softmax classifier in Faster RCNN is used for classification, the network cannot learn good features for classification. Therefore, Wang et al. used a cosine similarity-based classifier, which can learn classifiable features even in the case of small samples, reduce the variance between categories, and improve the classification accuracy.

[0054] This invention aims to select the optimal framework and optimal shot for fine-tuning-based few-shot steel defect detection through experiments. All experiments are based on the few-shot steel defect detection dataset of this invention, and ResNet50, ResNet101, and VGG16 within the Faster R-CNN framework, as well as YOLOv4, are selected for fine-tuning-based few-shot steel defect detection. YOLOv4 is chosen because it maintains higher accuracy and versatility across various scenarios compared to later versions such as YOLOv5, which lack these characteristics. Furthermore, this invention conducts comparative experiments on the number of shots under these frameworks, performing experiments with different backbones for 1-shot, 3-shot, 5-shot, 7-shot, and 10-shot samples. Specific results are shown in the table below, where all values ​​are mAP50.

[0055]

[0056] 1) The first conclusion from these controlled experiments is that regardless of the framework and backbone used, the number of shots has a significant impact on mAP. The mAP is lowest with 1 shot; increasing the number of shots increases mAP. However, observing the mAP of 7-shot and 10-shot models reveals that some models do not show significant changes in mAP, and some even outperform 10-shot models with 7-shot, indicating that the accuracy improvement between 7-shot and 10-shot models is not substantial. Furthermore, 7-shot is more in line with the principle of small sample size object detection than 10-shot. Therefore, overall, the model performs best when there are 7 samples per class.

[0057] 2) The second conclusion is that, by comparing the performance of different frameworks and backbones under 7-shot conditions, we found that the optimal model is the ResNet101 backbone using the Faster R-CNN framework. Compared to models using ResNet50 and VGG16 under the same framework, it outperforms by approximately 17% and 18% respectively, and compared to YOLOv4, it also outperforms by approximately 6%. Therefore, in the application scenario of this invention, the Faster R-CNN framework with a ResNet101 backbone is the optimal model, with an optimal mAP of 72.28%.

[0058] While feature pyramids can solve the problem of multi-scale object detection by passing semantic information from high-level features to lower levels, they also introduce the problem of spatial information loss. For example, some targets detected on small-scale feature maps may be occluded by targets detected on large-scale feature maps, leading to missing localization information and a decrease in detection accuracy. To address this issue, this invention incorporates a PAN (Path Aggregation Network) structure into the feature pyramid. This PAN structure introduces lateral connections and upsampling modules to pass semantic information from high-level features to lower levels, while simultaneously passing spatial information from lower-level features to higher levels, thereby improving the accuracy of object detection.

[0059] This invention proposes a novel hybrid domain attention network, SMSANet, or improved multi-scale attention network, by improving and optimizing the dual multi-scale attention network DMSANet. The specific structure of this attention network is as follows: Figure 2 As shown, the first module, the Split module, first divides the feature map into several groups, and uses the Shuffle unit to integrate channel attention and spatial attention into a block of each group. Sub-features are aggregated, and the "channel shuffle" operator is used to pass information between different sub-features.

[0060] The second module is the attention module, which consists of a sequential channel attention module and a spatial attention module. The attention module first passes through the channel attention module and then the spatial attention module.

[0061] Channel Attention Module: The input feature map of scale C*H*W is divided into 4 branches, from top to bottom: A, B, C, and D. After reshaping branches A and B, a feature map of scale C×N is generated. Then, the result of multiplying the transpose of branch B and branch A is subjected to a Softmax operation to generate a feature map of scale C×C. Then, the transpose of the feature map of scale C×C is multiplied with the reshaped C×N branch C, and then reshaped back to a feature map of size C*H*W and multiplied by a coefficient β. Finally, it is added to branch D to obtain a channel feature map that incorporates channel information.

[0062] Spatial Attention Module: First, the channel feature map of scale C*H*W is processed by three convolutional kernels to obtain three new feature maps, which are the feature maps of branches A, B, and C from top to bottom. Then, branches A and B are reshaped to generate feature maps of scale C×N. The result of multiplying the transpose of branches B and A is then processed by Softmax to generate a feature map of scale N×N. After that, the transpose of the N×N feature map is multiplied by the reshaped C branch, and then reshaped back to a feature map of size C*H*W and multiplied by a coefficient α. Finally, it is added to the original channel feature map to obtain a spatial feature map that incorporates spatial information.

[0063] After the two modules are executed sequentially, the final output is a feature map that integrates channel information and spatial information.

[0064] Through research and experimentation, the inventors finally proposed an improved network structure, such as... Figure 1 As shown, the input image first passes through the optimal backbone network of ResNet101 based on the Faster RCNN framework to obtain feature maps C2, C3, C4, and C5 at different scales, i.e., the initial multi-scale feature maps. Next, feature map C5 is input into an improved multi-scale attention network to obtain a feature map that integrates channel attention weights and spatial attention weights. This feature map is then added to feature map P5 after a 1×1 convolution. Subsequently, feature maps P4, P3, and P2 at different scales are obtained through top-down and lateral connections, i.e., intermediate multi-scale feature maps. Similarly, feature map P2 is passed through the improved multi-scale attention network and added to feature map P2 after a 1×1 convolution to obtain feature map F2. This feature map F2 is then obtained through bottom-up and lateral connections, i.e., the final multi-scale feature maps. Finally, these final multi-scale feature maps are fused and input into the RPN network, the ROIPooling layer, and the two branches of the classifier and regressor to obtain the final output result.

[0065] II. Training and Detection Network Model

[0066] A pseudo-label semi-supervised learning algorithm is used to train the detection network model. The local linear embedding algorithm is used to reduce the dimensionality and visualize the high-level features of the training output. Mislabeled or incorrectly labeled images are removed, and unlabeled image data is added. The training is then retrained to generate pseudo-labels, thereby expanding the training set until the training requirements are met.

[0067] Pseudo-label-based semi-supervised learning is a common method in semi-supervised learning. It involves using a model to predict unlabeled data and then using the predictions as pseudo-labels to train the model together with labeled data.

[0068] In pseudo-label-based semi-supervised learning algorithms, the accuracy of pseudo-labels significantly impacts model performance. Therefore, when using pseudo-labels, it's crucial to consider how to select high-quality ones. This could be achieved by setting a threshold to control confidence or by using regularization to reduce the impact of noise. This invention applies the Locally Linear Embedding (LLE) algorithm to improve pseudo-label-based semi-supervised learning algorithms. LLE is a non-linear dimensionality reduction algorithm capable of handling data with non-linear relationships and manifold structures. It can reduce dimensionality while preserving the local neighborhood structure of the data, thus better retaining the structure and information of the original data. Figure 3 This vividly illustrates the process of manifold dimensionality reduction.

[0069] This invention extracts high-level feature maps from the semi-supervised learning training phase, reduces these high-level feature maps to two dimensions using the LLE algorithm, and visualizes them using MATLAB. The visualization result is similar to the image shown. Finally, data cleaning is performed based on the feature distribution to remove low-quality mislabeled and incorrectly labeled data. Therefore, the final process of the pseudo-label semi-supervised learning algorithm improved by the LLE algorithm is as follows:

[0070] (1) Train the model using a small amount of labeled data.

[0071] (2) After training the initial model, extract high-level features, perform dimensionality reduction and visualization of the high-level features using the LLE algorithm, and then perform data cleaning to find mislabeled or incorrectly labeled images. After finding them, remove the labels and put in the unlabeled data.

[0072] (3) Use the trained model to predict the unlabeled data and use the prediction results as pseudo-labels.

[0073] (4) Combine the labeled data and pseudo-labels as an extended training set and continue training the model.

[0074] (5) Repeat steps 2 and 3 until the model converges and achieves the expected effect or reaches the preset number of training iterations to obtain the final model.

[0075] Small sample datasets need to adhere to the N-way K-shot principle; therefore, the small sample dataset of this invention also needs to be created according to this principle. N-way K-shot refers to a small sample learning method that allows the learning model to be trained on a limited sample set to obtain better generalization ability. Here, N represents the number of classes, and K represents the number of samples provided in each class. That is, the model needs to complete the task we define using these N*K samples, and it follows the requirements of Kang et al. for small sample datasets, where K must not exceed 10, that is, there cannot be more than 10 samples in each class.

[0076] The steel defect detection dataset studied in this invention originates from a data science competition on the Kaggle platform, provided by the Russian steel company Severstal. The dataset has been divided into training and testing sets for the competition. Statistical analysis of the training set revealed that it contains 12,568 images, of which 6,666 contain at least one defect, with a total of 7,095 defect annotations. These defects fall into four categories: dents, cracks, scratches, and iron oxide scale.

[0077] Visual analysis of defects in the steel dataset revealed significant differences in the size of different defects; for example, the size of iron oxide scale was tens of times larger than that of pits. Therefore, this invention conducts a detailed statistical study on the number of defect types and their pixel distribution.

[0078] First, scratches account for approximately 73% of the total, indicating a highly unbalanced distribution of defects in the dataset. Since scratches are the most numerous, they also represent the largest percentage of pixels, approximately 80%. In contrast, iron oxide scale accounts for only 11.3%, but comprises 16.8% of the pixels, suggesting that oxide scale is larger. Conversely, pits and cracks are much smaller, accounting for 12.6% and 3.48% of the total defects, respectively, but only 2.39% and 0.51% of the total mask pixel count. This suggests that these defects are very small, making it difficult for subsequent networks to detect pits and cracks.

[0079] Since the pixel scale of iron oxide scale is significantly larger than that of pits, clearer and more concise data is needed for illustration. Therefore, this invention has statistically analyzed the distribution of all defect pixels, and the specific data is shown in the table below.

[0080]

[0081] Based on the data in the chart, it is clear that pit defects have the smallest pixel scale, with a minimum pixel scale of 163. In contrast, cracks have an average pixel scale of 3378.4, making them the smallest of all defects. Scratches have the largest pixel scale, with a minimum pixel scale of 368240. In contrast, iron oxide scale has an average pixel scale of 34373.9, making it the largest of all defects. This indicates that the pixel scale range among the four types of defects in the dataset is quite large. Large-scale defects have large areas and rich features, making them generally easier to detect. However, small-scale defects have small areas, making identification and detection more difficult and placing higher demands on subsequent networks and network modules.

[0082] Fine-tuning the model requires a large number of base class samples and a small number of new class samples; therefore, the selection of base class defects is crucial. In this dataset, scratches are the base class with the most samples, totaling 5150, while dents are the second most numerous. Therefore, this invention defines scratches and dents as base class defects, while cracks and iron oxide scale are defined as new class defects. Considering the limitations of the fine-tuning model, only 10 new class samples are needed. Therefore, the small sample dataset of this invention consists of two base class defects and two new class defects, where N is 2 and the maximum value of K is 10. Through this method, this invention successfully transformed the steel defect dataset provided by Serverstal Steel Company into a small sample dataset based on fine-tuning.

[0083] III. Testing

[0084] The image to be inspected is input into a trained detection network model for recognition.

[0085] To verify the feasibility of the detection method of the present invention, we conducted the following experiments:

[0086] The first experiment was a small-sample steel defect detection experiment based on fine-tuning, incorporating the improved module of this invention. The experiment was conducted using an NVIDIA GeForce RTX 2070 GPU, and the network model was built using PyTorch, employing ResNet101 within the Faster R-CNN framework for defect detection. Training optimization used stochastic gradient descent with a batch size of 4, momentum of 0.9, and weight decay coefficient of 0.0001. The learning rate for the first stage of base class training was 0.02, and the learning rate for the second stage of new class training was 0.001. Then, a comparative experiment was conducted between the original network model and the improved network model based on the attention feature pyramid mechanism.

[0087] The second experiment focused on improving the model's accuracy after introducing an improved semi-supervised learning algorithm. The experiment used a complete steel defect dataset provided by Serverstal Steel. The LLE algorithm was used to clean 6000 training images, filtering out 143 training samples with mislabeled, incorrectly labeled, or missing labels. These samples were then added to the unlabeled dataset. A pseudo-labeled semi-supervised learning method was used to predict these images and add them to the training set to assist in the final model training.

[0088] Analysis of experimental results:

[0089] The improved few-sample object detection model based on fine-tuning in this invention makes full use of low-level and high-level semantic and spatial information, and the improved attention mechanism enables the final fusion of new features with rich information, thereby improving the accuracy of object detection. At the same time, the improved semi-supervised learning algorithm further improves the quality of the dataset labels in this study, making the accuracy of object detection even better.

[0090] 1. Single Defect Performance Analysis

[0091] This invention optimizes defect detection and verifies it through a 7-shot experiment (i.e., 7 samples per new class), with results as follows: Figure 4 As shown in the table below, the changes in the detection accuracy of four types of defects before and after the network model improvement are illustrated. The table shows that the detection accuracy of all defects was significantly improved in the improved network, with the greatest improvement in pit defects (5.34%), followed by crack defects (4.78%). Since these two types of defects are small targets, the improved model significantly improved their detection accuracy compared to the unimproved network. Furthermore, the improved model also reduced the false positives and incorrect detections of larger defects such as scratches (optimization level: 2.77%) and iron oxide scale (optimization level: 4.31%), further enhancing their detection accuracy.

[0092]

[0093] 2. Overall performance comparison and analysis of the models

[0094] To address the issue of the large pixel scale span among the four types of steel defects in the steel defect dataset—that is, the simultaneous existence of small and large target defects—this invention proposes an improved attention mechanism feature pyramid. Comparative experiments using small-sample target detection based on fine-tuning yield the following results: Figure 5 As shown, it is evident that after adding the attention mechanism and the feature pyramid mechanism, the mAP under 7-shot (i.e., the number of samples for each new class is 7) is improved by 2.34%. In addition, with the improved serial attention feature pyramid mechanism of this study, the mAP under 7-shot is improved by 4.3%.

[0095] To further verify the effectiveness of the improved scheme of this invention, the modules were separated and related separation experiments were conducted. The mAP results of small sample target detection under 7-shot were obtained after adding different modules, as shown in the table below. The results show that the modules of this invention can all help improve the accuracy of the base model. Considering that pits and cracks in steel defects are small target defects and the pixel scale span of the four types of defects is large, the model using FPN improved the accuracy by 0.89% compared to the base model, and the model using the FPN+DMSANet module improved the accuracy by 2.34% compared to the base model. Due to the defects of FPN, the model using the improved FPN improved the accuracy by 2.65% compared to the base model. After using the improved FPN+SMSANet, the model improved the performance by 4.3% compared to the base model, thus verifying the effectiveness of the improved modules of this invention.

[0096]

[0097] 3.4.2.2.3 Analysis of the Results of the Improved Semi-Supervised Learning Algorithm

[0098] During the training of the detection network in this invention, we discovered that some unlabeled data was not used in the first stage of base class training. Considering that in practical applications, obtaining high-quality labels often requires significant human and financial resources, and that steel defects are a difficult-to-label target category, we decided to adopt and improve upon a pseudo-label-based semi-supervised learning algorithm. To better clean the dataset, we used the LLE algorithm to reduce the dimensionality of 6047 training images in the pit and scratch dataset from the base class training. In the dimensionality-reduced dataset, we selected a total of 159 images suspected of being mislabeled, incorrectly labeled, or missing labels. These samples were added to the 1801 unlabeled images in the dataset to better enrich the diversity of the dataset and predict their pseudo-labels, thereby assisting in improving our model training. The final experimental results are shown in the table below.

[0099] We randomly sampled 1000-6000 training data points from dents and scratches at intervals of 1000 labeled data points. These samples, along with 1960 pseudo-labeled unlabeled data points predicted by a pseudo-label-based semi-supervised learning algorithm, were then fed into the network for model training. The results in the table show that the pseudo-label-based semi-supervised learning algorithm can transform previously useless unlabeled data into useful data, significantly improving few-shot detection algorithms. When all data was used for training, mAP increased from 76.58% to 78.21%. This demonstrates that in industrial applications where data is difficult to label and obtain, semi-supervised learning algorithms can utilize some unlabeled data, contributing to improved accuracy of the final model.

[0100]

[0101] 4. Model Detection Visualization Results

[0102] This invention employs a small-sample steel defect detection method based on an improved serial attention feature pyramid mechanism, achieving a final detection model accuracy of 78.21%. Next, we will visualize the detection results of this invention's model. First, we distinguish each defect by color: green represents dents, yellow represents cracks, purple represents scratches, and red represents iron oxide scale. The visualization effect of each defect is shown in Figure 6. Figure 6(a) and 6(b) In Figure 6(d), it can be clearly seen that the network detected the pits and cracks of the small target. As can be seen from Figure 6(d), the network also successfully detected the iron oxide scale of the large target. In addition, the scratches in Figure 6(c) were also successfully detected by the network. Figure 7 The image shows a visualization of two defects occurring simultaneously. The left image shows a defect image with both dents and cracks, while the right image shows a defect image with both scratches and iron oxide scale. Figure 8 The visualization shows the effect of three defects appearing simultaneously, including dents, cracks, and scratches.

[0103] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and various changes or modifications can be made to these embodiments without departing from the principles and essence of the present invention.

Claims

1. A small-sample steel defect detection method based on attention feature pyramid mechanism, characterized in that: I. Constructing a Detection Network Model First, an initial multi-scale feature map is obtained through a convolutional neural network. Then, an improved multi-scale attention network is used to process the initial multi-scale feature map one by one using a top-down and lateral connection approach to obtain intermediate multi-scale feature maps. Next, an improved multi-scale attention network is used to process the intermediate multi-scale feature maps one by one using a bottom-up and lateral connection approach to obtain the final multi-scale feature map. Finally, the final multi-scale feature maps are fused and sequentially input into the RPN network, ROI pooling layer, classifier, and regressor to output the final classification result. II. Training and Detection Network Model The detection network model is trained using a pseudo-label semi-supervised learning algorithm. The high-level features of the training output are reduced in dimensionality and visualized using the local linear embedding algorithm. Mislabeled images are removed and replaced with unlabeled image data. The training is then retrained to generate pseudo-labels, thereby expanding the training set until the training requirements are met. III. Testing The image to be inspected is input into a trained detection network model for recognition. The improved multi-scale attention network includes a Split module, as well as a serial channel attention module and a spatial attention module. The Split module is used to extract features from multi-scale feature maps and aggregate them into an input feature map with a scale of C*H*W; The channel attention module is used to divide the input feature map of size C*H*W into four branches, namely A, B, C, and D from top to bottom. After reshaping branches A and B, a feature map of size C×N is generated. Then, the result of multiplying the transpose of branch B and branch A is subjected to a Softmax operation to generate a feature map of size C×C. Then, the transpose of the feature map of size C×C is multiplied with the reshaped branch C of size C×N, and then reshaped back to a feature map of size C*H*W and multiplied by a coefficient β. Finally, it is added to branch D to obtain a channel feature map that incorporates channel information. The spatial attention module first uses three convolutional kernels to generate three new feature maps from the channel feature map of size C*H*W, which are the feature maps of branches A, B, and C from top to bottom. Then, branches A and B are reshaped to generate a feature map of size C×N. The result of multiplying the transpose of branches B and A is then subjected to a Softmax operation to generate a feature map of size N×N. The transpose of the N×N feature map is then multiplied with the reshaped C branch, and the result is reshaped back to a feature map of size C*H*W and multiplied by a coefficient α. Finally, this feature map is added to the original channel feature map to obtain a spatial feature map that incorporates spatial information.

2. The small-sample steel defect detection method based on the attention feature pyramid mechanism according to claim 1, characterized in that: The convolutional neural network uses the ResNet101 residual network.

3. The small-sample steel defect detection method based on the attention feature pyramid mechanism according to claim 2, characterized in that: First, the input image is processed through the optimal backbone network of ResNet101 to obtain feature maps C2, C3, C4, and C5 at different scales, which are the initial multi-scale feature maps. Next, feature map C5 is input into an improved multi-scale attention network to obtain a feature map that integrates channel attention weights and spatial attention weights. This feature map is then added to feature map P5 after a 1×1 convolution. Subsequently, feature maps P4, P3, and P2 at different scales are obtained through top-down and lateral connections, which are the intermediate multi-scale feature maps. Similarly, feature map P2 is processed through the improved multi-scale attention network and added to feature map P2 after a 1×1 convolution to obtain feature map F2. This feature map F2 is then obtained through bottom-up and lateral connections, which are the output feature maps at different scales, which are the final multi-scale feature maps. Finally, these final multi-scale feature maps are fused and input into the RPN network, the ROIPooling layer, and the two branches of the classifier and regressor to obtain the final output result.

4. The small-sample steel defect detection method based on the attention feature pyramid mechanism according to claim 1, characterized in that... Training the detection network model includes the following steps: (1) A pseudo-label semi-supervised learning algorithm is adopted, and a detection network model is trained using labeled small sample image data; (2) Extract the high-level features of each training result image, perform dimensionality reduction and visualization processing on the high-level features through the Local Linear Embedding (LLE) algorithm, then perform data cleaning, filter out mislabeled images and remove the labels, and put in unlabeled image data. (3) Use the trained detection network model to predict unlabeled image data and use the prediction results as pseudo-labels; (4) Combine the labeled small sample image data and pseudo-labels as an expanded training set to continue training the detection network model; (5) Repeat steps 2 and 3 until the detection network model converges and achieves the expected effect, or reaches the preset number of training times, to obtain the final detection network model.

Images