Small sample steel defect detection method based on attention feature pyramid mechanism

Abstract

A small sample steel defect detection method based on an attention feature pyramid mechanism, comprising: first acquiring initial multi-scale feature maps by means of a convolutional neural network, then by means of a top-down and transverse connection mode, using an improved multi-scale attention network to process the initial multi-scale feature maps one by one to obtain intermediate multi-scale feature maps, then by means of a bottom-up and transverse connection mode, using the improved multi-scale attention network to process the intermediate multi-scale feature maps one by one to obtain final multi-scale feature maps, and finally outputting a final classification result; using a pseudo-label semi-supervised learning algorithm to train a detection network model, using a locally linear embedding algorithm to perform dimension reduction and visualization processing on high-level features of training output results, screening for mistakenly labeled and incorrectly labeled images and removing labels, then putting the images into unlabeled image data, and performing re-training to generate pseudo labels so as to expand a training set until the training requirement is met; and inputting an image to be detected into a trained detection network model for identification.

Description

Small sample steel defect detection method based on attention feature pyramid mechanism Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence and relates to a small sample steel defect detection method based on an attention feature pyramid mechanism. Background Art

[0002] The importance of steel surface quality, especially cold-rolled steel, has become increasingly important since the 1980s, primarily due to demand from automotive manufacturers. Over time, the surface quality of hot strip steel, and therefore steel itself, has become increasingly important. However, during the steel production process, various defects can occur on the steel surface due to environmental factors, human factors, and other factors. Therefore, steel defect detection is a must-do post-production step.

[0003] Currently, most steel mills rely on human quality inspectors to detect steel defects. However, due to eye fatigue, traditional visual inspection methods are inefficient and have a high rate of missed detections. Furthermore, different inspectors have inconsistent defect detection standards, which also affects the effectiveness of steel defect detection. With the development of artificial intelligence (AI), AI can now achieve detection accuracy exceeding that of humans. Furthermore, machines are not constrained by external conditions such as the environment and climate, and can operate longer and at lower costs than humans. Therefore, machine vision technology, a field of AI, has been gradually applied to the industrial sector.

[0004] Traditional machine vision inspection first manually extracts features from images. However, steel defects are usually relatively small, and the naked eye is prone to errors when working for a long time. Feature extraction is also difficult, so the labor cost of feature extraction is high and the efficiency is low. In addition, we are now in the era of big data, and the amount of industrial image data that needs to be inspected is huge. Traditional machine vision is not suitable for processing big data. These problems make it difficult for traditional machine vision to meet the conditions of industrial applications in terms of accuracy and robustness.

[0005] Nowadays, deep learning detection has gradually replaced traditional machine vision detection. The advantage of deep learning is automatic feature extraction. Compared with manual feature extraction, deep learning is not only efficient and low-cost, but also more accurate in understanding sample features. Therefore, the accuracy of deep learning is also higher than traditional machine vision. However, deep learning detection also requires a large number of samples to ensure sufficient accuracy. However, it is difficult to collect enough useful samples on a real steel production line. Therefore, defect detection methods that require a large number of labeled samples cannot be directly applied to steel surface defect detection.

[0006] In view of the above, samples of steel surface defects are few and difficult to collect, and in the real world there is an urgent need to detect steel surface defects with few samples.

[0007] Summary of the Invention

[0008] The purpose of the present invention is to overcome the technical defects of traditional deep learning algorithms for steel plate surface defect detection, such as requiring a large number of samples to ensure sufficient accuracy. A small-sample steel defect detection method based on an attention feature pyramid mechanism is provided, which is optimized based on a fine-tuned small-sample target detection method. Modules such as the feature pyramid, attention mechanism and semi-supervised learning algorithm are improved, innovated and optimized to improve the accuracy and robustness of steel defect detection.

[0009] The present invention can be achieved through the following technical solutions:

[0010] A small sample steel defect detection method based on attention feature pyramid mechanism, including

[0011] 1. Building a detection network model

[0012] First, the initial multi-scale feature map is obtained through the convolutional neural network. Then, the initial multi-scale feature map is processed one by one by the improved multi-scale attention network in a top-down and lateral connection manner to obtain the intermediate multi-scale feature map. Then, the intermediate multi-scale feature map is processed one by one by the improved multi-scale attention network in a bottom-up and lateral connection manner to obtain the final multi-scale feature map. Finally, the final multi-scale feature map is fused and sequentially input into the RPN network, ROI pooling layer, classifier and regressor to output the final classification result.

[0013] 2. Training the Detection Network Model

[0014] A pseudo-label semi-supervised learning algorithm is used to train the detection network model. The high-level features of the training output are reduced in dimension and visualized using a local linear embedding algorithm. Mislabeled images are screened out and their labels are removed. Unlabeled image data is then added and retrained to generate pseudo-labels. This is used to expand the training set until the training requirements are met.

[0015] 3. Testing

[0016] The image to be inspected is input into the trained detection network model for recognition.

[0017] Furthermore, the improved multi-scale attention network includes a Split module and a serial channel attention module and a spatial attention module.

[0018] The Split module is used to extract features of multi-scale feature maps and aggregate them into an input feature map of scale C*H*W;

[0019] The channel attention module is used to divide the input feature map of scale C*H*W into 4 branches, which are A, B, C, and D branches from top to bottom. After reshaping, branches A and B generate a feature map of scale C×N. The result of multiplying the transpose of branch B and branch A is subjected to a Softmax operation to generate a feature map of scale C×C. The transpose of the feature map of scale C×C is then multiplied by the C branch reshaped to C×N, and then reshaped back to a feature map of size C*H*W and multiplied by a coefficient β. It is then added to the D branch to obtain a channel feature map that integrates channel information.

[0020] The spatial attention module is used to first obtain three new feature maps by passing the channel feature map of scale C*H*W through three convolution kernels, which are the feature maps of branches A, B, and C from top to bottom. The A branch and the B branch are then reshaped to generate a feature map of scale C×N. The result of multiplying the transpose of the B branch and the A branch is subjected to a Softmax operation to generate a feature map of scale N×N. The transpose of the feature map of scale N×N is then multiplied by the C branch reshaped to C×N, and then reshaped back to a feature map of size C*H*W and multiplied by a coefficient α. The result is then added to the original channel feature map to obtain a spatial feature map that integrates spatial information.

[0021] Furthermore, the convolutional neural network adopts ResNet101 residual network.

[0022] Furthermore, first, the input image will pass through the optimal backbone network of ResNet101 to obtain feature maps C2, C3, C4, and C5 of different scales, namely the initial multi-scale feature map; next, the feature map C5 will be input into the improved multi-scale attention network to obtain a feature map that integrates the channel attention weight and the spatial attention weight, and it will be added to the feature map C5 that has undergone 1×1 convolution to obtain the feature map P5. Subsequently, through top-down and horizontal connections, feature maps P4, P3, and P2 of different scales, namely the intermediate multi-scale feature map, are obtained; similarly, the feature map P2 will pass through the improved multi-scale attention network and be added to the feature map P2 that has undergone 1×1 convolution to obtain the feature map F2, and then through bottom-up and horizontal connections, feature maps of different scales are output, namely the final multi-scale feature map; finally, these final multi-scale feature maps are fused and input into the RPN network, ROI Pooling layer, classifier and regressor branches to obtain the final output result.

[0023] Furthermore, training the detection network model includes the following steps:

[0024] (1) Using pseudo-label semi-supervised learning algorithm, the detection network model is trained using labeled small sample image data;

[0025] (2) Extract high-level features of each training result image, perform dimensionality reduction and visualization on the high-level features through the local linear embedding algorithm (LLE), and then perform data cleaning to filter out mislabeled images and remove labels, and then add unlabeled image data;

[0026] (3) Use the trained detection network model to predict the unlabeled image data and use the prediction results as pseudo labels;

[0027] (4) Combine the labeled small sample image data and pseudo labels as an extended training set to continue training the detection network model;

[0028] (5) Repeat steps 2 and 3 until the detection network model converges and achieves the expected effect, or reaches the preset number of training times to obtain the final detection network model. Beneficial effects:

[0029] 1. Construction of a small sample dataset based on semi-supervised learning. To reduce reliance on manually labeled data, pseudo-label semi-supervised learning methods can significantly address this issue. Finally, the steel defect dataset is divided into steel defect categories according to the small sample standard, and the dataset is divided into base classes and new classes in a 1:1 ratio to construct a small sample dataset.

[0030] 2. Research on small-sample target detection based on fine-tuning methods. Due to the high precision requirements for steel surface defect detection in real life, this paper selected a small-sample learning method based on fine-tuning from common small-sample learning methods. Based on this, we conducted experimental comparisons with basic target detection frameworks, selected the framework with the highest accuracy, and then improved upon it.

[0031] 3. Small target detection algorithm based on feature fusion and attention mechanism. First, to address the problem of small steel defects with large spans that cannot be detected, feature fusion or feature pyramid modules are added to reduce structural information loss. Second, to address the problem of false detection and missed detection of small target defects, an attention module is added to enable the model to focus on the characteristics of small target defects in steel, thereby improving algorithm accuracy. Finally, by optimizing the semi-supervised learning algorithm and assisting model training, the model accuracy is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 is a schematic diagram of the overall process of the present invention;

[0033] FIG2 is a schematic diagram of the structure of the improved multi-scale attention network of the present invention;

[0034] FIG3 is an abstract schematic diagram illustrating the dimensionality reduction process performed by the LLE algorithm of the present invention;

[0035] FIG4 is a schematic diagram showing a comparison of single defect detection results when using the small sample steel defect detection method of the present invention and the existing method;

[0036] FIG5 is a schematic diagram showing a comparison of the results of steel defect detection using different detection models;

[0037] Figures 6(a), 6(b), 6(c), and 6(d) are schematic diagrams showing visualization results of pit defects, crack defects, scratch defects, and iron oxide scale defects when a small sample steel defect detection method of the present invention is used for detection;

[0038] FIG7 is a schematic diagram showing the visualization results of two coexisting defects when a small sample steel defect detection method of the present invention is used for detection;

[0039] FIG8 is a schematic diagram showing the visualization results of the coexistence of three defects when the small sample steel defect detection method of the present invention is used for detection. DETAILED DESCRIPTION

[0040] The specific implementation methods of the present invention will be further described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0041] In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0042] As shown in Figure 1, the present invention proposes a small-sample steel defect detection method based on the attention feature pyramid mechanism, optimizes the small-sample target detection method based on fine-tuning, and improves, innovates and optimizes modules such as the feature pyramid, attention mechanism and semi-supervised learning algorithm to improve the accuracy and robustness of steel defect detection.

[0043] The details are as follows:

[0044] 1. Building a detection network model

[0045] First, the initial multi-scale feature map is obtained through a convolutional neural network, and then the initial multi-scale feature map is processed one by one by the improved multi-scale attention network in a top-down and lateral connection manner to obtain an intermediate multi-scale feature map. Then, the intermediate multi-scale feature map is processed one by one by the improved multi-scale attention network in a bottom-up and lateral connection manner to obtain the final multi-scale feature map. Finally, the final multi-scale feature map is fused and input into the RPN network, ROI pooling layer, classifier and regressor in sequence to output the final classification result.

[0046] After the present invention studied and compared the small sample target detection methods, it was found that the target positioning accuracy of the method based on metric learning was not accurate enough; the idea of ​​the method based on data enhancement is to expand the small sample data set into a conventional target detection data set. The subsequent experimental steps are the same as conventional target detection, which cannot meet the actual situation and research objectives of current steel surface defect detection; when dealing with complex tasks, the generalization ability of the meta-model of the method based on meta-learning may be limited, resulting in poor performance. Therefore, the present invention finally decided to adopt a small sample target detection method based on fine-tuning. This method can achieve higher detection accuracy and meet the actual situation and research objectives of current steel surface defect detection.

[0047] Fine-tuning based few-shot object detection was originally proposed by Kang et al., which mainly provides few-shot learning capabilities by making full use of detection training data from base classes and quickly adjusting the detection prediction network based on a few supporting examples to predict new classes.

[0048] The model is divided into two phases. The first phase is to pre-train a base model on a large-scale annotated training set, such as the commonly used Faster R-CNN and YOLO. This base model can be trained on large-scale datasets such as ImageNet and has good generalization capabilities. The second phase is fine-tuning on a small sample training set. Typically, a small sample training set contains only a few dozen to a few hundred annotated data points, and the type and amount of annotated data are very limited. In this phase, the model parameters need to be adjusted to adapt to the small sample dataset and improve the model's performance on the object detection task. During fine-tuning, strategies such as learning rate decay similar to those used in the first phase can be adopted to ensure model convergence and robustness.

[0049] The present invention introduces the fine-tuning based small-sample object detection method proposed by Wang et al., who adopted the widely used two-stage object detector Faster R-CNN as the basic detection model. The Faster R-CNN model includes a backbone network (such as ResNet, VGG16), a region proposal network RPN, and a two-layer fully connected (FC) subnetwork as a proposal-level feature extractor, as well as a classifier for classifying object categories and a regressor for predicting boundary coordinates.

[0050] The first stage is actually the same as other object detection frameworks. The network is trained on a large number of samples, namely base class samples, and the loss function used is as follows: L = L rpn +L cls +L loc

[0051] Among them, L rpn Applied to the output of RPN to distinguish foreground and background and refine the anchors, L cls is the cross entropy loss of the box classifier C, L loc is the smoothed L1 loss of the box regressor R.

[0052] The second stage is the small sample fine-tuning stage. The dataset used includes a small number of base classes and new classes. First, the network assigns randomly initialized weights to the new classes, keeping the entire feature extractor fixed while only fine-tuning the classification and regression networks.

[0053] In the second stage, a classifier based on cosine similarity is used. Because in the case of small samples, if the traditional softmax classifier in Faster RCNN is used for classification, the network cannot learn good features for classification judgment. Therefore, Wang et al. use a classifier based on cosine similarity. In this way, even in the case of small samples, they can learn features that can be classified, reduce the variance between categories, and improve the classification accuracy.

[0054] The present invention aims to select the optimal framework and optimal shot for small-sample steel defect detection based on fine-tuning through experiments. All experiments are based on the small-sample steel defect detection dataset of the present invention, and ResNet50, ResNet101 and VGG16 under the Faster RCNN framework, as well as YOLOv4 are selected for small-sample steel defect detection based on fine-tuning. Among them, the reason for choosing YOLOv4 is that it can maintain higher accuracy and versatility in a variety of scenarios compared to subsequent versions such as YOLOv5, while subsequent versions do not have these features. In addition, the present invention conducted a control test on the number of shots under these frameworks, and conducted experiments on 1-shot, 3-shot, 5-shot, 7-shot and 10-shot under different backbones. The specific results can be seen in the table below, and the values ​​in the figure are all mAP50.

[0055] 1) The first conclusion drawn from these control experiments is that, regardless of the framework and backbone used, the impact of the number of shots on mAP is obvious. The mAP is lowest with 1-shot, and increases with increasing shots. However, when observing the mAP of 7-shot and 10-shot models, we found that the mAP of some models did not change significantly, and some even performed better with 7-shot than with 10-shot, indicating that the accuracy of the model will not be significantly improved with 7-shot and 10-shot. In addition, 7-shot is more consistent with the principle of small-sample object detection than 10-shot. Therefore, overall, the model performs best when there are 7 samples in each class.

[0056] 2) The second conclusion is that by comparing the 7-shot performance of different frameworks and backbones, we found that the optimal model is the ResNet101 backbone using the Faster RCNN framework. Compared to the ResNet50 and VGG16 models using the same framework, it outperforms them by approximately 17% and 18%, respectively, and also outperforms YoloV4 by approximately 6%. Therefore, in the application scenario of this invention, the Faster RCNN framework with a ResNet101 backbone is the optimal model, achieving an optimal mAP of 72.28%.

[0057] Although the feature pyramid can solve the problem of multi-scale object detection and pass the semantic information of high-level features to the lower levels, it also brings a problem, that is, the loss of spatial information. For example, some targets detected on the small-scale feature map may be blocked by the targets detected on the large-scale feature map, which will lead to the loss of positioning information and thus the decrease in detection accuracy. To solve this problem, the present invention adds a PAN (Path Aggregation Network) structure on the basis of the feature pyramid. The PAN structure introduces lateral connections and upsampling modules on the basis of the feature pyramid, passes the semantic information of high-level features to the lower levels, and at the same time passes the spatial information of low-level features to the higher levels, thereby improving the accuracy of target detection.

[0058] By improving and optimizing the dual multi-scale attention network DMSANet, this paper proposes a new hybrid domain attention network SMSANet, namely the improved multi-scale attention network. The specific structure of the attention network is shown in Figure 2. It includes the first module, namely the Split module, which first divides the feature map into several groups and uses the Shuffle unit to integrate channel attention and spatial attention into a block in each group. The sub-features are aggregated, and the "channel shuffle" operator is used to transfer information between different sub-features.

[0059] The second module is the attention module, which is a serial channel attention module and a spatial attention module. It first passes through the channel attention module and then through the spatial attention module.

[0060] Channel attention module: The input feature map of scale C*H*W is divided into 4 branches, namely A, B, C, and D branches from top to bottom. Branch A and branch B are reshaped to generate a feature map of scale C×N. The result of multiplying the transpose of branch B and branch A is subjected to a Softmax operation to generate a feature map of scale C×C. The transpose of the feature map of scale C×C is then multiplied by the C branch reshaped to C×N, and then reshaped back to a feature map of size C*H*W and multiplied by a coefficient β. The result is then added to the D branch to obtain a channel feature map that integrates channel information.

[0061] Spatial attention module: First, the channel feature map of scale C*H*W is passed through three convolution kernels to obtain three new feature maps, which are the feature maps of branches A, B, and C from top to bottom. Then, the A branch and the B branch are reshaped to generate a feature map of scale C×N. The result of multiplying the transpose of the B branch and the A branch is subjected to a Softmax operation to generate a feature map of scale N×N. After that, the transpose of the feature map of scale N×N is multiplied by the C branch reshaped to C×N, and then reshaped back to a feature map of size C*H*W and multiplied by a coefficient α. It is then added to the original channel feature map to obtain a spatial feature map that integrates spatial information.

[0062] After the serial processing of the above two modules, the final output is a feature map that combines channel information and spatial information.

[0063] After research and experiments, the inventors finally proposed an improved network structure, as shown in Figure 1. First, the input image will pass through the optimal backbone network of ResNet101 under the Faster RCNN framework to obtain feature maps C2, C3, C4, and C5 of different scales, namely the initial multi-scale feature map. Next, feature map C5 is input into the improved multi-scale attention network to obtain a feature map that integrates channel attention weights and spatial attention weights. It is added to the feature map C5 that has undergone 1×1 convolution to obtain feature map P5. Subsequently, through top-down and horizontal connections, feature maps P4, P3, and P2 of different scales are obtained, namely intermediate multi-scale feature maps. Similarly, feature map P2 is passed through the improved multi-scale attention network and added to the feature map P2 that has undergone 1×1 convolution to obtain feature map F2. Then, through bottom-up and horizontal connections, feature maps of different scales are output, namely the final multi-scale feature map. Finally, these final multi-scale feature maps are fused and input into the RPN network, ROI Pooling layer, classifier, and regressor branches to obtain the final output result.

[0064] 2. Training the Detection Network Model

[0065] A pseudo-label semi-supervised learning algorithm is used to train the detection network model. The local linear embedding algorithm is used to reduce the dimensionality and visualize the high-level features of the training output results. The mislabeled images are screened out and the labels are removed. Unlabeled image data is added and retrained to generate pseudo labels to expand the training set until the training requirements are met.

[0066] Pseudo-label based semi-supervised learning is a common method in semi-supervised learning. It uses the model to predict unlabeled data, and then uses the prediction results as pseudo labels to train the model together with labeled data.

[0067] In the semi-supervised learning algorithm based on pseudo-labels, the accuracy of the pseudo-labels has an important impact on the performance of the model. Therefore, when using pseudo-labels, it is necessary to consider how to screen out high-quality pseudo-labels, such as by setting a threshold to control the confidence level, or by using a regularization method to reduce the impact of noise, etc. The present invention applies the Locally Linear Embedding (LLE) algorithm to improve the semi-supervised learning algorithm based on pseudo-labels. The LLE algorithm is a nonlinear dimensionality reduction algorithm that can process data with nonlinear relationships and manifold structures, and can reduce dimensionality while maintaining the local neighborhood structure of the data, so that the structure and information of the original data can be better preserved. Figure 3 shows the process of manifold dimensionality reduction.

[0068] The present invention extracts high-level feature maps from the semi-supervised learning training phase, reduces these high-level feature maps to two dimensions using the LLE algorithm, and visualizes them using MATLAB. The visualization results are similar to the graph. Finally, the data is cleaned based on the distribution of features, and low-quality mislabeling and mislabeling are eliminated. Therefore, the final process of the pseudo-label semi-supervised learning algorithm improved by the LLE algorithm is as follows:

[0069] (1) Use a small amount of labeled data to train the model.

[0070] (2) After training the initial model, high-level features are extracted and the high-level features are reduced in dimension and visualized using the LLE algorithm. Data cleaning is then performed to find the mislabeled images, remove the labels after finding them, and put in the unlabeled data.

[0071] (3) Use the trained model to predict the unlabeled data and use the prediction results as pseudo labels.

[0072] (4) Combine the labeled data and pseudo labels as an extended training set and continue training the model.

[0073] (5) Repeat steps 2 and 3 until the model converges and achieves the expected effect or reaches the preset number of training times to obtain the final model.

[0074] Small sample datasets need to follow the N-way K-shot principle, so the small sample datasets of the present invention also need to be produced according to this principle. N-way K-shot refers to a small sample learning method that allows the learning model to be trained on a limited set of samples to obtain better generalization capabilities, where N represents the number of categories and K represents the number of samples provided in each category. That is, the model needs to complete the defined task using these N*K samples. In addition, it follows the requirements of Kang et al. for small sample datasets K-shot, where K must not exceed 10, that is, there cannot be more than 10 samples in each category.

[0075] The steel defect detection dataset studied in this paper originated from a data science competition on the Kaggle platform and was provided by the Russian steel company Severstal. The dataset was divided into training and test sets for the competition. Statistics from the training set showed that it contained 12,568 images, 6,666 of which contained at least one defect. A total of 7,095 defect annotations were collected, covering four types of defects: pits, cracks, scratches, and scale.

[0076] Visual analysis of defects in steel datasets revealed significant variations in the sizes of different defects. For example, the size of iron oxide scale is dozens of times larger than that of pits. Therefore, the present invention conducted a careful statistical study of the number of defect types and their pixel distribution.

[0077] First, the number of scratches accounts for approximately 73% of the total, indicating an extremely unbalanced proportion of defects in the dataset. Since scratches are the most numerous, their pixel scale also accounts for the highest proportion, approximately 80%. In comparison, scale types account for only 11.3%, but they account for 16.8% of the pixel scale, indicating that scale is large in size. Pits and cracks are small, accounting for 12.6% and 3.48% of the total defects, respectively, but in the pixel count of the defect mask, they only account for 2.39% and 0.51% of the total mask, respectively. This indicates that the scale of these defects is very small, and subsequent networks may have difficulty detecting pit defects and crack defects.

[0078] Since the pixel size of iron oxide scale is obviously larger than that of pits, clearer data is needed to illustrate it. To this end, the present invention has conducted statistics on the distribution of all defective pixels, and the specific data is shown in the following table.

[0079] According to the data in the chart, it is clear that the pit defect is the defect with the smallest pixel scale, with the smallest pit defect having a pixel scale of 163. In contrast, the average pixel scale of the crack is 3378.4, which is the smallest average pixel scale of all defects. The scratch defect is the defect with the largest pixel scale, with the largest scratch defect having a pixel scale of 368240. In contrast, the iron oxide scale has an average pixel scale of 34373.9, which is the largest average pixel scale of all defects. From this, it can be concluded that the pixel scales of the four defects in the dataset span a large range. Large-scale defects have large areas and rich features, so they are generally easier to detect. However, small-scale defects have small areas, are more difficult to identify and detect, and place higher requirements on subsequent networks and network modules.

[0080] The fine-tuning model requires a large number of base class samples and a small number of novel class samples. Therefore, the selection of base class defects is very important. In this dataset, scratches are the base class with the most defect samples, with a total of 5150 samples, and pits are the base class defects with the second most samples. Therefore, the present invention defines scratches and pits as base class defects, and cracks and iron oxide scales as novel class defects. Considering the limitations of the fine-tuning model, only 10 novel class samples are needed to meet the requirements. Therefore, the small sample dataset of the present invention consists of two base class defects and two novel class defects, where N is 2 and the maximum value of K is 10. Through such a production method, the present invention successfully produced the steel defect dataset provided by Serverstal Steel Company into a small sample dataset based on fine-tuning.

[0081] 3. Testing

[0082] The image to be inspected is input into the trained detection network model for recognition.

[0083] In order to verify the feasibility of the detection method of the present invention, we conducted the following experiments:

[0084] The first experiment, based on fine-tuning for small-sample steel defect detection, introduced the improved module of this invention. The experiment was conducted using an NVIDIA GeForce RTX 2070 GPU. The network model was built using PyTorch, employing ResNet101 within the Faster RCNN framework for defect detection. Training optimization used a stochastic gradient descent algorithm with a batch size of 4, a momentum of 0.9, and a weight decay coefficient of 0.0001. The learning rate for the first phase of base class training was 0.02, and the learning rate for the second phase of new class training was 0.001. The original network model was then compared with the improved network model based on the attention feature pyramid mechanism.

[0085] The second experiment examined the accuracy improvement of the model after introducing an improved semi-supervised learning algorithm. Using a complete steel defect dataset provided by Serverstal Steel, the experiment used the LLE algorithm to clean 6,000 training images. A total of 143 training samples with mislabeling, mislabeling, or missing labels were screened and added to the unlabeled dataset. Pseudo-label semi-supervised learning was then used to predict these data and add them to the training set to assist in training the final model.

[0086] Analysis of test results:

[0087] The improved fine-tuning-based small-sample target detection model of the present invention makes full use of the semantic and spatial information of low-level and high-level layers, and the improved attention mechanism finally integrates new features with rich information, thereby improving the accuracy of target detection. At the same time, the improved semi-supervised learning algorithm further improves the quality of the labels of the dataset of this project, making the target detection accuracy better.

[0088] 1. Single defect performance analysis

[0089] The present invention optimizes defect detection and verifies it through a 7-shot experiment (i.e., the number of samples for each new class is 7). The results are shown in Figure 4. The table below reflects the changes in the detection accuracy of the four defects before and after the network model is improved. It can be seen from the table that in the improved network, the detection accuracy of all defects has been significantly improved, among which the degree of optimization of pit defects is the largest (5.34%), followed by crack defects (4.78%). Since these two defects are small targets, the improved model has significantly improved its detection accuracy compared to the unimproved network. At the same time, the improved model can also reduce the false detection and misdetection ratio of scratches (optimization degree: 2.77%) and iron oxide scale (optimization degree: 4.31%), which are considered "large targets", further improving its detection accuracy.

[0090] 2. Comparative analysis of overall model performance

[0091] To address the wide pixel scale span of the four steel defects in the steel defect dataset, where both small and large defects coexist, this paper proposes an improved attention mechanism feature pyramid. A comparative test of small-sample object detection based on fine-tuning yields the results shown in Figure 5. Clearly, the addition of the attention and feature pyramid mechanisms improves mAP by 2.34% under 7-shot detection (i.e., the number of samples for each new class is 7). Combined with the improved serial attention feature pyramid mechanism in this study, mAP under 7-shot detection improves by 4.3%.

[0092] In order to further verify the effect of the improved solution of the present invention, the modules were separated and related separation experiments were carried out, and the mAP result values ​​of small sample target detection under 7-shot after adding different modules were obtained, as shown in the following table. The results show that the modules of the present invention can help improve the accuracy of the basic model. In view of the characteristics that pits and cracks in steel defects are small target defects and the pixel scale span of the four defects is large, the model after adopting FPN is improved by 0.89% compared with the basic model, and the model after adopting FPN+DMSANet module is improved by 2.34% compared with the basic model. Due to the defects of FPN, the model after adopting the improved FPN is improved by 2.65% compared with the basic model. After adopting the improved FPN+SMSANet, the performance of the model is improved by 4.3% compared with the basic model, which finally verifies the effectiveness of the improved module of the present invention.

[0093] 3.4.2.2.3 Analysis of the Improved Semi-Supervised Learning Algorithm

[0094] During the detection network training process of the present invention, we found that some unlabeled data were not used in the first stage of base class training. Taking into account that in actual application scenarios, the cost of having high-quality labels often requires manpower and financial resources, and steel defects are difficult to label target types, we decided to adopt a semi-supervised learning algorithm based on pseudo-labels and improve and optimize it. In order to better clean the dataset, we used the LLE algorithm to reduce the dimension of 6047 training data in the pit and scratch dataset in the base class training. In the dataset after dimensionality reduction, we screened out a total of 159 pictures that were suspected to be mislabeled, wrongly labeled, and missing, and added these samples to 1801 unlabeled unlabeled datasets to better enrich the diversity of the dataset and predict their pseudo-labels, thereby assisting in improving our model training. The final experimental results are shown in the table below.

[0095] We randomly sampled 1,000 to 6,000 training data points from pits and scratches at intervals of 1,000 labeled data points. This data, along with 1,960 pseudo-labeled unlabeled data points predicted by a semi-supervised learning algorithm based on pseudo-labels, was fed into the network for model training. The results in the table show that the pseudo-label-based semi-supervised learning algorithm can transform previously useless unlabeled data into useful data, significantly improving the small-sample detection algorithm. When all data were used in training, the mAP increased from 76.58% to 78.21%. This demonstrates that in industrial fields where data labeling is difficult and challenging to obtain, the use of semi-supervised learning algorithms can leverage some of the unlabeled data, helping to improve the accuracy of the final model.

[0096] 4. Model detection visualization results

[0097] The present invention uses a small-sample steel defect detection method based on an improved serial attention feature pyramid mechanism, and the resulting detection model has an accuracy of 78.21%. Next, we will visualize the detection results of the detection model of the present invention. First, we distinguish each defect by color: green represents pits, yellow represents cracks, purple represents scratches, and red represents iron scale. The visualization effect of each defect is shown in Figure 6. In Figures 6(a) and 6(b), it can be clearly seen that the network detects pits and cracks on small targets. From Figure 6(d), it can be seen that the network also successfully detects iron scale on large targets. In addition, the network also successfully detects the scratches in Figure 6(c). Figure 7 shows the visualization effect of an image with two defects appearing simultaneously. The left image is a defect image with both pits and cracks, and the right image is a defect image with both scratches and iron scale. Figure 8 shows the visualization effect of an image with three defects appearing simultaneously, including pits, cracks, and scratches.

[0098] Although specific embodiments of the present invention have been described above, those skilled in the art will appreciate that these are merely examples and that various changes or modifications may be made to these embodiments without departing from the principles and essence of the present invention.

Claims

1. A small sample steel defect detection method based on an attention feature pyramid mechanism, characterized in that:

1. Building a detection network model First, the initial multi-scale feature map is obtained through a convolutional neural network, and then the initial multi-scale feature map is processed one by one by an improved multi-scale attention network in a top-down and lateral connection manner to obtain an intermediate multi-scale feature map. Then, the intermediate multi-scale feature map is processed one by one by an improved multi-scale attention network in a bottom-up and lateral connection manner to obtain a final multi-scale feature map. Finally, the final multi-scale feature map is fused and input into the RPN network, ROI pooling layer, classifier and regressor in turn to output the final classification result.

2. Training the Detection Network Model The detection network model is trained using a pseudo-label semi-supervised learning algorithm. The high-level features of the training output are reduced in dimension and visualized using a local linear embedding algorithm. Mislabeled images are screened out and their labels are removed. Unlabeled image data is then added for retraining to generate pseudo-labels, thereby expanding the training set until the training requirements are met.

3. Detection The image to be inspected is input into the trained detection network model for recognition.

2. The small sample steel defect detection method based on the attention feature pyramid mechanism according to claim 1 is characterized in that: The improved multi-scale attention network includes a Split module and a serial channel attention module and a spatial attention module. The Split module is used to extract features of multi-scale feature maps and aggregate them into input feature maps with a scale of C*H*W; The channel attention module is used to divide the input feature map of scale C*H*W into 4 branches, namely A, B, C, and D branches from top to bottom. Branch A and branch B are reshaped to generate feature maps of scale C×N. The result of multiplying the transpose of branch B and branch A is subjected to Softmax operation to generate a feature map of scale C×C. Then, the feature map of scale C×C is generated. After multiplying the transpose of the C branch reshaped to C×N, it is reshaped back to a feature map of size C*H*W and multiplied by a coefficient β, and then added to the D branch to obtain a channel feature map that integrates channel information; The spatial attention module is used to first obtain three new feature maps through three convolution kernels for the channel feature map of scale C*H*W, which are the feature maps of branches A, B, and C from top to bottom. Then, the A branch and the B branch are reshaped to generate a feature map of scale C×N. Then, the result of multiplying the transpose of the B branch and the A branch is subjected to a Softmax operation to generate a feature map of scale N×N. Then, the transpose of the feature map of scale N×N is multiplied by the C branch reshaped to C×N, and then reshaped back to a feature map of size C*H*W and multiplied by a coefficient α, and then added to the original channel feature map to obtain a spatial feature map that integrates spatial information.

3. The small sample steel defect detection method based on the attention feature pyramid mechanism according to claim 2 is characterized in that: The convolutional neural network adopts ResNet101 residual network.

4. The small sample steel defect detection method based on the attention feature pyramid mechanism according to claim 3 is characterized in that: First, the input image will pass through the optimal backbone network of ResNet101 to obtain feature maps C2, C3, C4, and C5 of different scales, namely the initial multi-scale feature map; next, the feature map C5 will be input into the improved multi-scale attention network to obtain a feature map that integrates the channel attention weights and spatial attention weights, and it will be added to the feature map C5 that has undergone 1×1 convolution to obtain the feature map P5. Subsequently, through top-down and lateral connections, feature maps P4, P3, and P2 of different scales, namely the intermediate multi-scale feature map, are obtained; similarly, the feature map P2 will be passed through the improved multi-scale attention network and then added to the feature map P2 that has undergone 1×1 convolution to obtain the feature map F2, and then through bottom-up and lateral connections, feature map outputs of different scales, namely the final multi-scale feature map, are obtained; finally, these final multi-scale feature maps are fused and input into the RPN network, ROI Pooling layer, and the two branches of the classifier and regressor to obtain the final output result.

5. The small sample steel defect detection method based on the attention feature pyramid mechanism according to claim 1 is characterized in that Training the detection network model includes the following steps: (1) Using pseudo-label semi-supervised learning algorithm, the detection network model is trained using labeled small sample image data; (2) Extract high-level features of each training result image, perform dimensionality reduction and visualization on the high-level features through the local linear embedding algorithm (LLE), and then perform data cleaning to filter out mislabeled images and remove labels, and then put in unlabeled image data; (3) Use the trained detection network model to predict the unlabeled image data and use the prediction results as pseudo labels; (4) The labeled small sample image data and pseudo labels are combined as an extended training set to continue training the detection network model; (5) Repeat steps 2 and 3 until the detection network model converges and achieves the expected effect, or reaches the preset number of training times to obtain the final detection network model.

Images