Strip steel surface defect multi-view-field detection method based on semi-supervised learning

Abstract

The invention discloses a strip steel surface defect multi-illumination field-of-view detection method based on semi-supervised learning. The method comprises the following steps: S1, collecting pictures; s2, manual labeling is carried out; s3, training the model; s4, predicting a pseudo label; and S5, repeating the steps S3 and S4. According to the method, the problem of low algorithm accuracy caused by mutual confusion of partial defects due to defect identification in a traditional single view field is solved.

Description

Technical Field

[0001] The present invention relates to a strip steel visual inspection technology, and more specifically to a strip steel surface defect multi-field inspection method based on semi-supervised learning. Background Art

[0002] Strip surface defect detection technology is a research hotspot in the field of metallurgical industry. Timely and accurate detection of surface defects can improve product quality and production efficiency and reduce economic losses. In the production process of strip steel, CCD cameras are generally used to collect strip surface images, and then machine vision is used to detect and classify defects. Deep learning algorithms, which are neural networks derived from machine learning, have dominated the important academic direction of computer vision in recent years. However, in the field of industrial vision, the application of deep learning technology is still in its early stages. For strip surface defect detection, the huge workload brought by data labeling tasks and the low detection accuracy caused by the single feature of defect images are important reasons why deep learning technology is not suitable for this application scenario. In recent years, industrial cameras with multi-field illumination (bright field, dark field, side field, etc.) have provided richer visual information for surface defect detection due to the multi-illumination field images they provide. The changes in light and dark features under different illumination fields are often the key factors in identifying and distinguishing certain defects.

[0003] Light sources with different fields of view and installation angles of industrial cameras Figure 1 As shown in the figure, the main difference lies in the different angles of the light source and the camera relative to the material to be tested: the bright field light source is reflected by the surface of the material to be tested and then directly enters the camera sensor (direct or scattered). At this time, the entire background of the material surface will be completely illuminated, and defects such as color difference and dirt will be clearly visible; the dark field light source has a smaller angle with the material to be tested, and after diffuse reflection, it partially enters the camera sensor. At this time, the material background image is darker, and some defects may be partially illuminated due to the change in the reflection angle of the light source; the side field light source is incident at a certain angle from the side of the material in the width direction, and the reflected light source does not directly enter the camera sensor, so the image background is very dark. However, when there are scratch defects on the surface of the material in the length direction, the side field light source will make it appear, showing a bright line in the image. Therefore, the multi-field images formed by light sources at different angles can restore the true appearance of material defects to the greatest extent, providing a more comprehensive and three-dimensional basis for judgment for the algorithm.

[0004] like Figure 2 As shown in the figure, by observing pictures from different fields of view, the same defect presents completely different characteristics. When identifying a defect, it may be difficult to distinguish using only a single field of view, but by observing pictures from multiple fields of view, different defect categories can be easily distinguished. Therefore, if the detection algorithm only uses a single field of view for defect identification, it is likely to cause some defects to be confused with each other, resulting in low algorithm accuracy and difficulty in meeting the needs of on-site use.

[0005] Existing patented technologies are mainly based on deep learning methods, such as:

[0006] Chinese patent application No. 201910899204.6 "A method for detecting surface defects of hot-rolled strip steel based on deep learning", 201910899204.6 "A method for detecting surface defects of hot-rolled strip steel based on deep learning", 202010002806.X "A method and device for detecting surface defects of hot-rolled strip steel based on deep learning", 202210113743.4 "A method for detecting surface defects of strip steel based on improved YOLOv5 network" and 202111365464.9 "A method for detecting defects of hot-rolled strip steel based on improved Faster R-CNN" each disclose a method for detecting surface defects of strip steel based on manual data annotation and improved deep learning models (YOLOv3, YOLOv5, Cascade RCNN and Faster RCNN, etc.). Although the above patent technologies have improved the classic models, they cannot solve the problems of large workload of data annotation and single defect image features.

[0007] Chinese patent application No. 202111058596.7 "A method for detecting slip defects on the surface of strip steel based on deep learning" and No. 202011569750.2 "A method for detecting defects on strip steel based on deep learning" both proposed a solution for defect detection by semantic segmentation of the surface image of strip steel. However, the semantic segmentation method of the above patented technology is only effective for some defects with complete and clear contours, and is powerless for large-area diffuse chromatic aberration defects.

[0008] Chinese patent application No. 202210952650.0 "An Intelligent Detection Method for Strip Defects" discloses a defect detection method that determines the grayscale image segmentation threshold of the strip surface based on a fuzzy S-type membership function. Although this patented technology improves the detection effect of pitting defects on the strip surface by improving the threshold segmentation method, it is less practical for strip varieties with many types of defects and complex and changeable shapes.

[0009] Chinese patent application No. 201410352867.3 "A strip defect detection method based on Gaussian filter multi-resolution filtering" proposes a method of using Gaussian filter to perform multi-resolution filtering on strip images, obtain strip differential images of different resolutions, and detect surface defects through threshold segmentation after fusion. However, this patent technology only involves defect detection and does not solve the problem of defect classification.

[0010] Chinese patent application No. 202211056662.1 "A method for classifying surface defects of strip steel" proposes to increase the number of defect images and enrich the diversity of defect features by constructing a generative adversarial network generator and discriminator, so as to solve the overfitting problem caused by the lack of surface defect images of hot-rolled strip steel in industrial environments. However, this patented technology can only increase the amount of labeled data, and cannot fundamentally solve the problem of single defect image features.

[0011] Chinese patent application No. 202211039714.4 "A strip surface defect detection method and system" proposes to use each strip surface image captured by each lighting field (light source / angle) as an independent channel feature map, and annotate each channel feature map with defects to form a defect training sample. The channel feature maps are fused to train a strip surface defect classification model. However, this patented technology only realizes the classification of strip defects, and does not provide a defect detection frame, so it is impossible to obtain key information such as the specific location and size of the defect.

[0012] Through the above patented technology, it was found that the existing strip surface defect detection method has problems such as large labeling workload, single defect image features, low detection accuracy, and low versatility. Summary of the invention

[0013] In view of the defects existing in the prior art, the purpose of the present invention is to provide a multi-field-of-view detection method for strip surface defects based on semi-supervised learning, which solves the problem of traditional single-field-of-view defect identification, which causes confusion between some defects and leads to low algorithm accuracy.

[0014] To achieve the above object, the present invention adopts the following technical solution:

[0015] A method for detecting surface defects of strip steel using multiple illumination fields of view based on semi-supervised learning, comprising the following steps:

[0016] S1. Collect pictures;

[0017] S2, manual labeling;

[0018] S3, training model;

[0019] S4, predict pseudo labels;

[0020] S5. Repeat steps S3 and S4.

[0021] Preferably, the step S1 specifically includes:

[0022] Collect images of all types of defects and all lighting fields from different production line units to form a defect detection data set.

[0023] Preferably, the number of the pictures is N×S;

[0024] Where N is the number of illuminated fields of view, and S is the number of images in each illuminated field of view.

[0025] Preferably, the illuminated field of view includes a bright field, a dark field and a side field.

[0026] Preferably, the step S2 specifically includes:

[0027] All the images in the defect detection data set are grouped according to the illumination field, and a group of images of the main illumination field n1 are selected, a total of S images, which are manually annotated in proportion, the location of the defect is framed and the category is marked, and the annotated data is set to L.

[0028] Preferably, the proportion of the pictures that are manually annotated is 5% to 10%.

[0029] Preferably, the step S3 specifically includes:

[0030] The labeled data L is used to train a target detection model.

[0031] Preferably, the step S4 specifically includes:

[0032] The trained target detection model is used to perform inference prediction on the remaining unlabeled images to generate pseudo-labels. After removing the erroneous pseudo-labels, the remaining pseudo-labels are incorporated into the label data L.

[0033] Preferably, the step S3 of training the target detection model using the labeled data L specifically includes the following steps:

[0034] S31, channel stacking the corresponding pictures in different illumination fields;

[0035] S32, selecting the target detection model, and changing the channel parameter of the input part of the target detection model to N;

[0036] S33. In the encoder stage of the target detection model, a multi-field attention module is added after the ordinary convolution residual module; after the target detection model is trained, the parameters of the target detection model are saved.

[0037] S34, testing the detection effect of the model on new defect image data. Repeat steps S32 to S34 to adjust the training setting parameters.

[0038] Preferably, the step S31 specifically includes:

[0039] Suppose a picture s in group n1 i , whose size is 1×H×W, and all corresponding images in other illumination field groups are sequentially combined with si Stack in the channel direction to get a picture s of size N×H×W i ;

[0040] All images of different illumination fields are stacked in the above manner, and finally a total of S images of size N×H×W are obtained, which finally constitute the defect detection data set;

[0041] Among them, 1 is the number of channels of a black and white image, H and W are the image height and width respectively, and N is the number of channels.

[0042] Preferably, in step S32, the target detection model is a Faster RCNN series or a YOLO series.

[0043] The present invention provides a multi-light field detection method for strip surface defects based on semi-supervised learning. The semi-supervised method can save a lot of manual labeling time and improve the efficiency of defect detection project development; the multi-light field detection technology stacks defect images of different light fields through channels to achieve light field feature fusion, and strengthens the multi-field feature extraction capability of the model by adding a multi-field attention module to improve the detection effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of different illumination fields;

[0045] Figure 2 It is a schematic diagram of defects appearing under different lighting fields;

[0046] Figure 3 It is a schematic flow chart of a method for detecting surface defects of steel strips using multiple illumination fields of view according to the present invention;

[0047] Figure 4 Schematic diagram of defect marking in step S2 of the method for detecting surface defects of steel strips by multiple illumination fields of view according to the present invention;

[0048] Figure 5 Schematic diagram of channel stacking in step S31 of the method for detecting surface defects of steel strips with multiple illumination fields of view according to the present invention;

[0049] Figure 6 It is a schematic diagram of adding a multi-field attention module in step S33 of the multi-light field detection method for strip surface defects of the present invention, (a) is the existing residual structure, and (b) is the multi-field attention residual structure. DETAILED DESCRIPTION

[0050] In order to better understand the above technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0051] Combination Figure 3As shown, the present invention also provides a multi-light field detection method for strip surface defects based on semi-supervised learning, comprising the following steps:

[0052] S1. Collect pictures: Collect N×S pictures of all types of defects in all illumination fields from different production line units (assuming there are N illumination fields, each with S pictures) to form a defect detection data set.

[0053] The illumination field includes bright field, dark field and side field.

[0054] S2. Manual labeling: Group all images in the defect detection dataset according to the illumination field, select a group of images with the main illumination field (such as bright field) n1, a total of S images, and manually label 5% to 10% of the images, select the location of the defect and mark the category, such as Figure 4 As shown, let the labeled data be L.

[0055] S3. Training model: Use the labeled data L to train the target detection model (such as Faster RCNN, YOLO, etc.).

[0056] S4. Predict pseudo labels: Use the trained target detection model to perform inference and prediction on the remaining unlabeled images to generate pseudo labels (the format is the same as manual labeling). After removing erroneous pseudo labels, the remaining pseudo labels are incorporated into the labeled data L.

[0057] S5. Repeat the loop process of steps S3 and S4 until all images are labeled, that is, L=S.

[0058] In each cycle of training the target detection model and predicting pseudo-labels, the amount of labeled data is rapidly expanded. The target detection model trained with more labeled data can provide more accurate pseudo-labels for the remaining unlabeled images. In practical applications, the above semi-supervised learning method can basically complete the data labeling of tens of thousands of images after 5 cycles. The subsequent model training is directly performed on all labeled data without the need to predict pseudo-labels.

[0059] In the above step S3, using the labeled data L to train the target detection model specifically includes the following steps:

[0060] S31, stack the corresponding images in different illumination fields through channels, such as Figure 5 As shown, the specific method is as follows:

[0061] Suppose a picture s in group n1 i , whose size is 1×H×W, and all corresponding images in other illumination field groups are sequentially combined with s i Stack in the channel direction to get a picture s of size N×H×W i ;

[0062] All images of different illumination fields are stacked in the above way, and finally a total of S images of size N×H×W are obtained, which finally constitute the defect detection data set;

[0063] Among them, 1 is the number of channels of a black and white image, H and W are the image height and width respectively, and N is the number of channels.

[0064] S32, select a target detection model (such as Faster RCNN series, YOLO series, etc.), and change the channel parameter of the input part of the target detection model to N to adapt to the input multi-illumination field of view data set.

[0065] S33. In the encoder stage of the target detection model, a multi-field attention module is added after the ordinary convolution residual module to help the target detection model better utilize the defect features of different illumination field channels, such as Figure 6 Therefore, during the training process, let the target detection model learn to adaptively focus on the key feature channels. After the target detection model training is completed, save the parameters of the target detection model.

[0066] S34, testing the detection effect of the model on new defect image data. Repeat steps S32 to S34, adjust the training setting parameters, optimize the training process, and obtain a better defect detection model.

[0067] The key technical points of the present invention are evaluated from the two aspects of development efficiency and detection performance:

[0068] First, the manual labeling time that can be saved by the semi-supervised learning method adopted by the present invention is estimated in a quantitative manner. Assume that the average time spent by the labeler to label a picture is t1 seconds. After the model prediction result is given, the average time for the labeler to check and remove erroneous pseudo-labels is t2 seconds. When the semi-supervised learning method is not adopted, the labeler needs to manually label all S pictures, and the total time consumed is T1=S*t1; after adopting the semi-supervised learning method, it is assumed that the labeler labels 10% of the pictures in step S2, and the remaining 90% of the pictures are pseudo-labeled by the model prediction, and then manually checked. At this time, the manual time consumed is T2=0.1*S*t1+0.9*S*t2 (the model training and testing process only requires simple settings, no manual intervention is required, and the manual time consumed can be ignored). It can be seen from the experience summarized in the practical process that the unit time t2 for the labeler to check the pseudo-label is approximately 1 / 10 of the unit time t1 for manual labeling. Under this assumption, the total manual time consumed by the semi-supervised method is T2 = 0.19*S*t1 = 0.19*T1. In other words, the semi-supervised learning method can help save more than 80% of the manual annotation time in the defect detection project development process, greatly improving the work efficiency of the development team.

[0069] Example 1

[0070] The hardware configuration used in this embodiment 1 is Intel Xeon (R) E5-2630 CPU processor, NVIDIA RTX1080Ti graphics card, and the software environment is CUDA10.2 and cuDNN9.2. The model is built using the PyTorch framework, and the specific steps are as follows:

[0071] Step 1: Collect about 3×40,000 images (3 illumination fields, each with about 40,000 images) of all types of defects in all illumination fields (bright field, dark field, and side field) on different production line units to form a defect detection data set.

[0072] Step 2: Divide all images in the defect detection dataset into three groups according to the illumination field, select a total of 40,000 bright field images, manually annotate approximately 3,000 of them, select the defect location and mark the category.

[0073] Step 3: Stack the three images corresponding to different illumination fields in the order of bright field, dark field, and side field in the channel direction to form a complete three-channel image (similar to the RGB three channels of natural images, where the three channels are bright, dark, and side fields). Construct a total of 40,000 three-field image datasets.

[0074] The entire dataset is randomly divided into a training set and a validation set in a ratio of 4:1.

[0075] Step 4: Select a deep learning detection model (such as Faster RCNN series, YOLO series, etc.), and change the channel parameter of the model input part to 3 to adapt to the input multi-field dataset. In the encoder stage of the model, add a multi-field attention convolution module after the ordinary convolution residual module (Faster RCNN's encoder ResNet and YOLO series encoder CSPNet are both composed of basic unit residual modules, so multi-field attention modules can be added) to build a multi-field detection model.

[0076] Step 5: Train the multi-field detection network model on the training set image data and save the model parameters. The trained model performs inference prediction on the remaining unlabeled images to obtain pseudo-labels for the remaining images.

[0077] Step 6: Remove incorrect pseudo-annotations and incorporate the remaining correct pseudo-annotations into manual annotations.

[0078] Step 7: Repeat steps 5 and 6 until all images are labeled.

[0079] Step 8: Train the multi-scene detection model on the labeled multi-scene image dataset.

[0080] Step 9: Test the detection effect of the model on the steel coil image data actually collected at the production site. Repeat step 8, adjust the training parameters, such as learning rate, training batch data size, training rounds, input image size, optimizer type, data enhancement method, etc., train the model and verify the effect, find the best training configuration and save the optimal model parameters.

[0081] Example 2

[0082] This Example 2 demonstrates the effect of improving the strip defect detection accuracy of the multi-field detection technology proposed by the present invention through online experimental results. When applying the algorithm, two major indicators generally need to be considered: real-time and accuracy. Real-time requires that the algorithm output detection results can be roughly synchronized with strip production, without obvious lag; accuracy requires that the classification and positioning of the algorithm output detection results are consistent with manual experience, and there should not be too many false detections. In the implementation of this patent, the real-time requirement of the algorithm that the processing time of a single image shall not exceed 10ms, and the accuracy of the algorithm requires that the accuracy of the detection results (the proportion of defects accurately classified and positioned to all statistical defects) shall not be less than 85%.

[0083] Table 1 Experimental results of multi-field detection method

[0084]

[0085] The above table shows that the multi-illumination field detection method for strip surface defects of the present invention, including the fusion input of multiple field images and the addition of multiple field attention modules, can improve the defect detection effect of the network model while having little effect on the real-time performance of the model. In actual operation, the multi-field detection method shows extremely high practical value.

[0086] The present invention proposes a multi-illumination field-of-view detection method for strip surface defects, wherein the semi-supervised method can save a lot of manual labeling time and improve the efficiency of defect detection project development; the multi-field-of-view detection technology stacks channels of defect images in different fields of view to achieve field-of-view feature fusion, and enhances the multi-field-of-view feature extraction capability of the model by adding a multi-field attention module to improve the detection effect.

[0087] Those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention, and are not intended to limit the present invention. As long as they are within the spirit of the present invention, any changes or modifications to the above embodiments will fall within the scope of the claims of the present invention.

Claims

1. A method for detecting surface defects of strip steel using multiple illumination fields based on semi-supervised learning, characterized in that: The following steps are involved: S1. Collect pictures; S2, manual labeling; S3, training model; S4, predict pseudo labels; S5. Repeat steps S3 and S4.

2. The method for detecting surface defects of strip steel using multi-light fields of view based on semi-supervised learning according to claim 1, characterized in that: The step S1 specifically includes: Collect images of all types of defects and all lighting fields from different production line units to form a defect detection data set.

3. The method for detecting surface defects of strip steel using multiple illumination fields based on semi-supervised learning according to claim 2, characterized in that: The number of pictures is N×S; Where N is the number of illuminated fields of view, and S is the number of images in each illuminated field of view.

4. The method for detecting surface defects of strip steel using multiple illumination fields based on semi-supervised learning according to claim 3 is characterized in that: The illumination field includes a bright field, a dark field and a side field.

5. The method for detecting surface defects of strip steel using multiple illumination fields based on semi-supervised learning according to claim 3 is characterized in that: The step S2 specifically includes: All the images in the defect detection data set are grouped according to the illumination field, and a group of images of the main illumination field n1 are selected, a total of S images, which are manually labeled in proportion, the location of the defect is framed and the category is marked, and the labeled data is set to L.

6. The method for detecting surface defects of steel strips based on semi-supervised learning in multiple illumination fields according to claim 5, characterized in that: The proportion of the pictures that are manually annotated is 5% to 10%.

7. The method for detecting surface defects of strip steel using multiple illumination fields based on semi-supervised learning according to claim 6, characterized in that: The step S3 specifically includes: The labeled data L is used to train a target detection model.

8. The method for detecting surface defects of steel strips using multiple illumination fields based on semi-supervised learning according to claim 7, characterized in that: The step S4 specifically includes: The trained target detection model is used to perform inference prediction on the remaining unlabeled images to generate pseudo-labels. After removing the erroneous pseudo-labels, the remaining pseudo-labels are incorporated into the label data L.

9. The method for detecting surface defects of steel strips using multiple illumination fields based on semi-supervised learning according to claim 7, characterized in that: In step S3, using the labeled data L to train the target detection model specifically includes the following steps: S31, channel stacking the corresponding pictures in different illumination fields; S32, selecting the target detection model, and changing the channel parameter of the input part of the target detection model to N; S33. In the encoder stage of the target detection model, a multi-field attention module is added after the ordinary convolution residual module; after the target detection model is trained, the parameters of the target detection model are saved. S34, testing the detection effect of the model on new defect image data. Repeat steps S32 to S34 to adjust the training setting parameters.

10. The method for detecting surface defects of steel strips using multiple illumination fields based on semi-supervised learning according to claim 9, characterized in that: The step S31 specifically includes: Suppose a picture s in group n1 i , whose size is 1×H×W, and all corresponding images in other illumination field groups are sequentially combined with s i Stack in the channel direction to get a picture s of size N×H×W i ; All images of different illumination fields are stacked in the above manner, and finally a total of S images of size N×H×W are obtained, which finally constitute the defect detection data set; Among them, 1 is the number of channels of a black and white image, H and W are the image height and width respectively, and N is the number of channels.

11. The method for detecting surface defects of steel strips by multiple illumination fields based on semi-supervised learning according to claim 9, characterized in that: In step S32, the target detection model is a Faster RCNN series or a YOLO series.

Images