Semi-supervised image detection method for steel surface defects

Abstract

The invention discloses a semi-supervised image detection method for steel surface defects, and relates to the technical field of image recognition. The method comprises the following steps: constructing a semi-supervised steel surface defect data set, and dividing the steel surface defect data set into a labeled steel surface defect data set and an unlabeled steel surface defect data set; the method is improved based on a YOLOv5 model, firstly, a feature extraction layer is improved, and an aggregation module is used in an original ResNet152 network model to replace a convolution block so as to enhance the feature extraction capability of the network. And secondly, an original dual-feature pyramid structure is improved in a feature learning layer by utilizing group space pyramid pooling, and the number of parameters is reduced through a group convolution method, so that the module is more efficient, and the detection precision of the model is improved. And finally, in a prediction layer, selecting a feature prediction region by means of an RPN network frame, selecting a prediction feature region in a multi-path parallel mode, calculating an error value between frames selected for multiple times, and correcting the prediction frame through regression variance, so that the reliability of pseudo tag generation is improved.

Description

Technical Field

[0001] This invention belongs to the field of image detection and recognition technology, specifically a semi-supervised image detection method for defects on steel surfaces. Background Technology

[0002] Steel, as a fundamental material, is widely used in national defense, military, and construction, thus requiring stringent quality standards. my country, as the world's largest producer and consumer of steel, accounts for more than half of global steel production and consumption annually. Steel production is frequently affected by factors such as raw materials, production methods, and manufacturing techniques, making product quality defects unavoidable. Surface defect detection is the most direct way to assess steel quality. To ensure steel quality and pass rates, surface defect detection is an essential and crucial step in the quality inspection process. Traditional detection methods primarily rely on manual inspection. With increasing production volume, manual inspection speed far exceeds production speed, resulting in unnecessary time consumption and hindering mass production. Furthermore, manual inspection is susceptible to subjective factors and experience, making it impossible to achieve uniform inspection standards. This paper compares two existing technological methods: machine vision-based inspection methods and deep learning-based inspection methods.

[0003] Machine vision-based inspection methods largely overcome the drawbacks of manual inspection. This method utilizes image processing technology to extract feature information from steel surface images and then performs defect location and identification based on the extracted feature information. However, some technical limitations remain. For example, the device has poor applicability and cannot accurately detect defects on steel surfaces with unusual shapes. For instance, Chinese patent application number CN202110681828.8 discloses an online measurement method for steel coil dimensions and an online detection method for omnidirectional end-face defects. This method uses multiple sensors and instrumental vision to measure the inner diameter, outer diameter, and thickness of the steel coil, effectively identifying end-face defects that include depth information. However, it cannot accurately determine the type of defect present in the steel and cannot accurately locate the defect.

[0004] With the development of deep learning, it has been widely applied in steel inspection. Deep learning-based methods have solved the shortcomings of traditional methods, such as poor performance and low efficiency. Furthermore, traditional image processing methods rely on manual evaluation of detection features each time, and these evaluation criteria include, but are not limited to, features such as shape, color, texture, and aspect ratio, inevitably leading to inconsistent detection results. For example, existing technology ("A Steel Surface Defect Detection Method Based on Improved Faster R-CNN", by Yang Li, 202107) discloses a method that introduces a guided anchor point candidate region network to predict the position and shape of anchor points, and designs an adjustable mechanism to solve the problem of network anchor point shape offset exceeding the region of interest, thereby mitigating the influence of irrelevant features. However, errors exist in the selection of feature regions at steel surface defects, resulting in insufficient accuracy in the prediction of steel surface defect types at the output layer.

[0005] Deep learning models are based on labeled data, but in actual manufacturing processes, detecting surface defects in steel faces several challenges: a limited number of defective steel products, complex defect types, and high labeling costs. Semi-supervised learning avoids the significant costs of labeling and retraining while ensuring model accuracy and reliability. To address the issues of inaccurate detection accuracy and high training costs in existing technologies, this paper proposes a semi-supervised image detection method for steel surface defects. Summary of the Invention

[0006] The purpose of this invention is to provide a semi-supervised image detection method for steel surface defects, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A semi-supervised image detection method for steel surface defects includes the following steps:

[0009] S10. Obtain the steel image dataset;

[0010] A set of steel images was obtained using the NEU-DET and NEU-CLS-64 datasets.

[0011] Furthermore, in order to train the steel surface defect detection model, steel surface defect images are selected from the steel image data to generate a training sample set.

[0012] S20. Construct the training set;

[0013] The steps include: preprocessing the images in the training set described in S21;

[0014] The preprocessing includes: mosaic data enhancement, adaptive anchor boxes, and adaptive scaling;

[0015] The mosaic data augmentation method randomly reduces large samples into small samples, increasing the number of small-scale targets.

[0016] The adaptive anchor frame is mainly used to locate the defect position in the steel image during the detection process;

[0017] The adaptive scaling scales the original input image to a fixed size before feeding it into the detection network;

[0018] S22. Define the bounding box of the surface defect region in the preprocessed image;

[0019] The steel surface defect region box is the specific region of the steel surface defect. Each region box represents a steel surface defect region, denoted as A = [x, y, w, h].

[0020] Where x is the horizontal coordinate of the upper left corner of the steel surface defect area frame, y is the vertical coordinate of the steel surface defect area frame, w is the width of the steel surface defect area frame, and h is the height of the steel surface defect area frame.

[0021] S23. Divide the training sample set into a labeled training sample set X = (x... c ,y c ): c∈(1,....,N) and unlabeled training sample set U=u c :c∈(1,...,M); where x c Let y represent the c-th steel surface defect image in the labeled training sample set. c This indicates the surface defect category corresponding to the c-th steel surface defect image, u c Let N represent the c-th steel surface defect image in the unlabeled training sample set, and let N and M represent the number of steel surface defect images in the labeled training sample set and the unlabeled training sample set, respectively, with N>M.

[0022] Furthermore, in order to accurately detect defects on the surface of steel, a detection model was established.

[0023] S30. Establish a steel surface defect detection model;

[0024] The steel surface defect detection model is an improvement based on the YOLOv5 model, which includes a feature extraction layer, a feature learning layer, and a prediction layer.

[0025] The feature extraction layer uses an improved ResNet152 network model to extract features from the steel surface defect images in the training sample set;

[0026] The features of the steel surface defect images in the training sample set mainly include: shape, texture, color, etc.

[0027] Furthermore, the improved ResNet152 network model includes an input layer, an aggregation module, a residual block, and an output layer.

[0028] The input layer takes the steel surface defect images in the training sample set as 152×152 pixels and uses a 1×1 convolution kernel to extract the steel surface defect images in the training sample set to obtain the first feature map;

[0029] Furthermore, the aggregation module effectively reduces the sampling density in the spatial domain, enabling the multi-scale architecture to have higher computational efficiency and capture a larger receptive field.

[0030] The aggregation module includes max pooling, 3×3 convolutional kernels, upsampling, and full connection. As the network progresses, the output channels of the aggregation module are typically increased until they reach the same resolution as the first feature map, thus obtaining the second feature map.

[0031] The improved ResNet152 network model, where each residual block consists of a series of aggregation modules, batch normalization layers, and activation functions.

[0032] In the residual block, the aggregation module and normalization perform feature extraction of the second feature map, adjust the number of input channels as skip connections through a 1×1 convolution operation, and activate the result using the Leaky ReLU activation function to obtain the output of the residual block.

[0033] The convolutional kernel in the output layer restores the dimensions of the second feature map to maintain the same dimensions as the first feature map, performs residual connections, and outputs the third feature map.

[0034] Furthermore, the feature learning layer is a multi-scale feature learning structure. By changing the spatial pyramid pooling layer of the original dual feature pyramid structure, the 3×3 convolutional blocks of the spatial pyramid pooling layer are replaced with 32 1×1 convolutional blocks. The third feature is input into the feature learning layer, and three feature maps of different sizes are obtained through the original YOLOv5 feature learning layer. The three feature maps of different sizes are then passed through the group of spatial pooling layers to obtain the same feature maps as the original three different sizes.

[0035] Meanwhile, during the process of sending the data to the prediction layer, the output of the original group space pyramid pooling layer and the feature maps of these three scales are joined by a residual connection to obtain the fourth feature map.

[0036] Furthermore, the prediction layer includes a Region Candidate Network (RPN) and a Region of Interest Pooling (Roi pooling).

[0037] The fourth feature map is convolved with a 1×1 kernel to unify the dimensions; the RPN generates candidate regions for the feature map; the feature map extracted from the region of interest is pooled to obtain the final predicted feature map, which is then used to predict the classification and bounding box results respectively.

[0038] S40. Train the steel surface defect detection model based on the training sample set, and obtain the steel surface defect detection model when the loss function value reaches the minimum.

[0039] Furthermore, the steel surface defect detection model is trained jointly by the labeled training set and the unlabeled training set, and the specific steps include:

[0040] S41. Use the labeled training sample set to train and initialize the steel surface defect detection model;

[0041] S42. Input the unlabeled training sample dataset into the initialized steel surface defect detection model, perform prediction, and label it with pseudo-labels;

[0042] S43. Train the initial steel surface defect detection model together with the pseudo-labeled training sample dataset and the labeled training sample set, optimize the parameters, and obtain the steel surface defect detection model.

[0043] The generation of the feature candidate regions affects the correctness of the pseudo-labels in the unlabeled dataset and the performance of the steel surface defect detection model. Therefore, a multi-path parallel selection method is used to select the feature map candidate regions, and the selection error before and after is calculated using the following formula:

[0044]

[0045]

[0046] Where τ is the average error value of the candidate region of the steel surface defect feature map, and μ k This is the derived value of the k-th candidate region of the steel surface defect feature map extracted by the dithering frame algorithm of the RPN network. It is μ k The normalized value, h(b) k ) and w(b k ) represent the height and width of the candidate region for the k-th steel surface defect feature map, respectively.

[0047] S50. Use the steel surface defect detection model to detect surface defects in the steel image.

[0048] S60. Output the type of steel surface defect and the area of ​​steel surface defect.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] 1. The feature extraction layer of the original YOLOv5 was replaced with an improved ResNet152 network model. The improved ResNet152 network model replaced the 3×3 convolutional blocks of the ResNet152 model with aggregation modules to enhance the network's feature extraction capabilities. Furthermore, it expands the receptive field during the extraction of defects from steel surfaces, effectively capturing subtle defect features and preserving more useful feature information for subsequent feature learning.

[0051] 2. This invention introduces a multi-scale feature learning structure into the feature learning layer, primarily by utilizing group spatial pyramid pooling to improve the original dual feature pyramid spatial pyramid pooling layer. Group spatial pyramid pooling reduces the number of parameters through group convolution, making the module more efficient. Furthermore, the group convolution method, similar to regularization, effectively reduces the risk of model fitting and improves the model's detection accuracy.

[0052] 3. This invention improves the prediction layer of the detection model, primarily by adjusting the RPN network's selection of candidate regions for steel surface defect features through a multi-path parallel selection method, thereby enhancing the accuracy of the final feature map prediction. Furthermore, it effectively improves the reliability of pseudo-label generation for samples in the unlabeled training set under semi-supervised conditions. Based on the original loss function, the difference between multiple candidate box selections is used as one of the evaluation criteria, thus altering the loss function and improving model performance. Attached Figure Description

[0053] Figure 1 A flowchart of a semi-supervised image detection method for steel surface defects provided in an embodiment of the present invention;

[0054] Figure 2 A diagram of steel surface defects provided for an embodiment of the present invention;

[0055] Figure 3 The aggregation module provided in this embodiment of the invention;

[0056] Figure 4 This is a group space pyramid pooling diagram provided in an embodiment of the present invention;

[0057] Figure 5 The model comparison results are shown in the embodiment of the present invention;

[0058] Figure 6 The detection result diagram provided for the embodiments of the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Please see Figures 1 to 5 A semi-supervised image detection method for steel surface defects is proposed, and the technical solution is as follows:

[0061] Specifically, refer to Figure 1 As shown in the flowchart, the present invention provides a semi-supervised image detection method for defects on the surface of steel, which includes steps S10 to S60 in a specific implementation.

[0062] Specifically, S10. Obtain a steel image dataset; S20. Construct a training set; S30. Establish a steel surface defect detection model; S40. Train the steel surface defect detection model based on the training sample set to obtain a steel surface defect detection model; S50. Use the steel surface defect detection model to detect surface defects in steel images; output the steel surface defect type and the steel surface defect region.

[0063] S10. Obtain the steel image dataset;

[0064] The steel image dataset was obtained through datasets NEU-DET and NEU-CLS-64;

[0065] Among them, the NEU-DET dataset is a steel surface defect dataset established by Northeastern University; the NEU-CLS-64 dataset is a dataset established by Northeastern University for steel surface defect image classification, with an image size of 64x64.

[0066] S20. Construct a training set by generating a training sample set based on the surface defect data in the steel image dataset;

[0067] Furthermore, in order to enrich the objects detected, the images in the training sample set are preprocessed.

[0068] The preprocessing method is as follows: The main principle of mosaic data augmentation is equivalent to inputting 4 images at once for learning during the training process, increasing the number of training samples and target diversity, improving the network training convergence speed and detection accuracy, and randomly reducing large samples into small samples, increasing the number of small-scale targets.

[0069] The adaptive anchor box calculation uses a genetic algorithm and a K-Means algorithm to calculate the optimal anchor boxes. The adaptive anchor box algorithm is mainly used to locate defects in steel images during the detection process. Its calculation process is as follows: Obtain the width and height of all targets in the image; scale each image proportionally to a specified size, ensuring that the maximum value of the width and height conforms to the specified size; change the bounding boxes from relative coordinates to absolute coordinates, multiplying by the scaled width and height; filter the bounding boxes, retaining those with a width and height greater than or equal to two pixels; use K-means clustering to obtain k (k>1) anchors; use a genetic algorithm to randomly mutate the width and height of the anchors. If the mutation is effective, the result is assigned to the anchors; if the mutation is ineffective, it is skipped, with a default mutation count of 1000. The fitness is calculated using the `anchor_fitness` method and then evaluated.

[0070] Adaptive image scaling is typically required for various object detection algorithms. This involves scaling the original input image to a fixed size before feeding it into the detection network. However, this traditional scaling method has drawbacks. In practice, many images have varying aspect ratios, resulting in inconsistent black border sizes after scaling and padding. Excessive padding leads to significant information redundancy, impacting overall inference speed. To further improve inference speed, an adaptive approach is needed to add minimal black borders to the scaled image.

[0071] The operation process is as follows:

[0072] (1) Calculate the scaling ratio based on the original image size and the size of the image input to the network;

[0073] (2) Calculate the size of the scaled image based on the original image size and the scaling ratio;

[0074] (3) Calculate the black border fill value;

[0075] In the implementation of this invention, the size of the input image is fixed at 512×512, and the surface defect area bounding box of the preprocessed image is marked.

[0076] Furthermore, in order to learn the steel surface defect detection model in a semi-supervised manner, the training sample set is divided as follows;

[0077] Specifically, the steel surface defects include six types: cracks, impurities, patches, pitting, indentation of iron oxide scale, and scratches. Images of these six types of steel surface defects are collected, such as... Figure 2 As shown.

[0078] Furthermore, the iron surface defect images are divided into a training set and a test set.

[0079] Specifically, the NEU-DET dataset and the NEU-CLS-64 dataset were used. The NEU-DET dataset contains 6 types of defects, with 300 images for each defect, totaling 1800 images. The NEU-CLS-64 dataset contains 9 types of surface defects without labels, totaling 7226 images. The steel image dataset was divided in a 3:7 ratio. The training sample set includes a labeled training sample set X = (x... c ,y c ): c∈(1,....,N) and unlabeled training sample set U=u c :c∈(1,...,M);

[0080] Where, x c Let y represent the c-th steel surface defect image in the labeled training sample set. c This indicates the surface defect category corresponding to the c-th steel surface defect image, u c Let N represent the c-th steel surface defect image in the unlabeled training sample set, and let N and M represent the number of steel surface defect images in the labeled training sample set and the unlabeled training sample set, respectively, with N>M.

[0081] S30. Establish a steel surface defect detection model;

[0082] The steel surface defect detection model is based on the YOLOv5 model, which includes a feature extraction layer, a feature learning layer, and a prediction layer.

[0083] In this embodiment, an improved ResNet152 network model is used as an enhancement to the feature extraction layer of the YOLOv5 model. The ResNet152 network model itself enhances network capacity, feature extraction, and generalization ability. By introducing an aggregation module, multiple channel convolutional groups of receptive fields are used to improve feature extraction capabilities. For steel surface defect features of different scales, it can effectively learn receptive field allocation strategies, preserving valuable feature information for the feature learning layer.

[0084] The feature extraction layer uses an improved ResNet152 network model to extract features from the steel surface defect images in the training sample set;

[0085] The features of the steel surface defect images in the training sample set mainly include: shape, texture, color, etc.

[0086] Furthermore, the improved ResNet152 network model includes an input layer, an aggregation module, residual blocks, and an output layer. The input layer takes the steel surface defect images in the training sample set as 152×152 pixels and extracts the first feature of the training sample set using 1×1 convolutional kernels. The aggregation module replaces multiple 3×3 convolutional blocks in the original ResNet152 network model, and the second feature map is obtained in the aggregation module. Figure 3 The structure of the aggregation module is shown in the figure, which includes max pooling, 3×3 convolution kernels, upsampling, and full connection.

[0087] As the network progresses forward, the output channels of the aggregation modules are typically increased until they match the resolution of the first feature map. The improved ResNet152 network model consists of a series of aggregation modules, batch normalization layers, and activation functions in each residual block. In the residual block, the input is processed by the aggregation modules and normalization layers to extract features. A 1×1 convolution operation is used to adjust the number of input channels as skip connections. The result is then activated using the LeakyReLU activation function to obtain the output of the residual block. The activation function used is the Leaky ReLU function, a variation of the ReLU function designed to solve the Dead ReLU problem. The function initializes neurons with a small value of 0.01, making ReLU more inclined to activate rather than vanish in negative regions. The function expression is:

[0088]

[0089] Where λ is a number that converges in the range (0,1);

[0090] In the output layer, the dimensions of the second feature map are restored using a 1x1 convolution kernel, maintaining the same dimensions as the first feature map, and residual connections are performed to obtain the third feature map.

[0091] Furthermore, experimental comparisons are presented in Table 1. Table 1 shows that the aggregation module used in this method is more effective in feature extraction of steel surface defects. Compared with the original ResNet152 model, the mAP50 of this method reaches 68.7%. Especially when the target steel surface defect is small, the mAP... s The mAP was 22.9%, while when using the original ResNet152 as the feature extraction network, the mAP was... s It is 18.4%.

[0092] In this embodiment, this step is improved in the feature learning layer, changing the spatial pyramid pooling in the original dual-feature pyramid structure to group spatial pyramid pooling. Figure 4As shown, the group space pyramid pooling reduces the number of parameters, thus making the model more efficient.

[0093] Furthermore, the feature learning layer is a multi-scale feature learning structure. By modifying the spatial pyramid pooling layer of the original dual feature pyramid structure, the 3×3 convolutional blocks of the spatial pyramid pooling layer are replaced with 32 1×1 convolutional blocks. The second feature is input into the feature learning layer, and after passing through the original YOLOv5 feature learning layer, three feature maps of different sizes are obtained. These three feature maps of different sizes are then passed through the group spatial pooling layer to obtain feature maps identical to the original three different sizes, and then passed in parallel through a 5x5 max pooling layer, a 9x9 pooling layer, and a 13x13 pooling layer. Simultaneously, during the process of sending the data to the prediction layer, the output of the original group spatial pyramid pooling layer and the feature maps of these three scales are residually concatenated to obtain a fourth feature map.

[0094] Furthermore, to illustrate the benefits of the proposed group spatial feature pooling layer, experimental comparisons were conducted. In Table 1, the "Improved ResNet152 + Dual Feature Pyramid + Multi-path Parallel Selection Method" retains the pooling portion of the original dual pyramid. Experiments showed that the mAP50 and ACC values ​​were 67.5% and 95.9%, respectively, while the mAP50 and ACC values ​​of this method were 68.7% and 96.8%. The experimental results demonstrate that using a multi-scale feature learning structure is more effective for feature learning.

[0095] Furthermore, in this embodiment, the step of selecting the predicted feature map using a multi-path parallel selection method based on the Region Candidate Network (RPN) and comparing the results of multiple selection regressions, and using the difference between the regression variances to correct and adjust the candidate box, can effectively reduce the error caused by the feature extraction process and further enhance the reliability of the dithered box.

[0096] Furthermore, the prediction layer includes a Region Candidate Network (RPN) and Region of Interest (ROI) pooling. Three feature maps of different sizes are output from the feature learning layer and passed through 1×1 convolutional kernels to unify their dimensions. The RPN generates candidate regions for the feature maps. Predicted feature maps are obtained by pooling the feature maps extracted from the ROIs and are used to predict classification and bounding box results, respectively.

[0097] Furthermore, the feature learning layer outputs three feature maps of different sizes, which are then processed by a 1×1 convolution kernel to unify the dimensionality. The RPN generates candidate regions for the feature maps, and the feature maps extracted from the regions of interest are pooled to obtain the final predicted feature maps, which are then used to predict the classification and bounding box results, respectively.

[0098] The steel surface defect type and steel surface defect region are predicted separately by fully connected components. The calculation formula for the steel surface defect region is as follows:

[0099] G x =P w d x (P)+P x +τ

[0100] G y =P h d y (P)+P y +τ

[0101] G w =P w exp(d w (P))+τ

[0102] G h =P h exp(d h (P))+τ

[0103] Where, P x P y P w and P h These are the x-coordinate, y-coordinate, height, and width of the center of the candidate region of the steel surface defect feature map, respectively; G x G y G h and G w Let x, y, height, and width be the center of the defect region on the steel surface.

[0104] Furthermore, by utilizing a multi-path parallel selection method to calculate the error of the candidate region for the steel surface defect feature map, the prediction of steel surface defect types is effectively improved, and errors in defect location selection are avoided. Table 1 shows a comparative experiment with the single-path selection method. The data shows that the multi-path parallel selection method has higher accuracy than the single-path selection method. The mAP value for small targets is 22.9%, and the mAP for medium and large targets is [not specified in the original text]. ml It is 32.6%.

[0105] S40. Train the steel surface defect detection model based on the training sample set to obtain the steel surface defect detection model;

[0106] Furthermore, the steel surface defect detection model is trained jointly by the labeled training set and the unlabeled training set; the specific steps include:

[0107] S41. Use the labeled training sample set to train and initialize the steel surface defect detection model;

[0108] S42. Input the unlabeled training sample dataset into the initialized steel surface defect detection model, perform prediction, and label it with pseudo-labels;

[0109]

[0110] in, Let M() represent the unlabeled training sample data, M() represent the predicted probability distribution of the initial steel surface defect detection model, and u c θ represents the c-th unlabeled image of steel surface defects, where θ is a hyperparameter.

[0111] S43. Train the initial steel surface defect detection model together with the pseudo-labeled training sample dataset and the labeled training sample set, optimize the parameters, and obtain the steel surface defect detection model;

[0112] The generation of the feature candidate regions affects the correctness of the pseudo-labels in the unlabeled sample dataset and the performance of the steel surface defect detection model. Therefore, a multi-path parallel selection method is used to select the feature map candidate regions, and the selection error before and after is calculated using the following formula:

[0113]

[0114]

[0115] Where τ is the average error value of the candidate region of the steel surface defect feature map, and μ k This is the derived value of the k-th candidate region of the steel surface defect feature map extracted by the dithering frame algorithm of the RPN network. It is μ k The normalized value, h(b) k ) and w(b k ) represent the height and width of the candidate region for the k-th steel surface defect feature map, respectively.

[0116] Furthermore, during the training process of the steel surface defect detection model, the performance of the model is improved by changing the loss function by using the difference between multiple candidate box selections as one of the evaluation criteria, based on the original loss function. The calculation formula is as follows:

[0117]

[0118] Where H represents cross-entropy, N and M represent the number of labeled training set samples and the number of unlabeled training set samples, respectively, and τ1 and τ2 represent the mean selection error of candidate boxes in the unlabeled sample dataset and the unlabeled sample dataset, respectively.

[0119] Further, repeat steps S10 to S40.

[0120] S50. Use the steel surface defect detection model to detect surface defects in the steel image.

[0121] S60. Output the type of steel surface defect and the area of ​​steel surface defect.

[0122] In this embodiment, to illustrate the optimal combination scheme of the method, the effectiveness of the method is demonstrated by comparative ablation experiments using a steel surface defect detection model. Figure 5 Table 1 shows the evaluation index values ​​for various combinations of this method, where, in Figure 5 The figure shows the detection accuracy values ​​obtained by each model after different number of tests. The first, second, third, and fourth models represent ResNet152+ multi-scale feature learning structure + multi-path parallel selection method, improved ResNet152+ dual feature pyramid + multi-path parallel selection method, improved ResNet152+ multi-scale feature learning structure + single-path selection method, and the proposed method, respectively. The figure shows that the proposed method achieves the highest ACC value after 100 tests, and at different stages... Figure 4 All of these indicate that this paper achieved the highest score.

[0123] Table 1. Comparison of ablation experiments on steel surface defect detection models

[0124]

[0125] Furthermore, to evaluate the performance of the detection model, this invention uses various evaluation metrics to compare with existing methods. Among them, mAP and FPS are basic evaluation metrics in detection. mAP is calculated from precision (Pr) and recall (Re), and the calculation formula is as follows:

[0126]

[0127]

[0128]

[0129] Wherein, TP, FP, and FN are respectively the number of actual labels and correctly predicted labels in the steel surface defect image, the number of steel surface defect images that do not contain labels but are predicted to have labels, and the number of steel surface defect images that contain labels but are not predicted, and c is the defect type.

[0130] This embodiment compares the performance of four methods in a semi-supervised scenario, using mAP and FPS as evaluation metrics. In the same operating environment, the batch size is set to 4, and 200 epocch sessions are trained. Finally, the corresponding evaluation metrics are generated, as shown in Table 2, where mAP 50% represents a threshold of 0.5. Figure 6 The table also shows some of the detection results. By comparing the proposed method with other methods, as shown in Table 2, it is verified that the method has excellent detection performance compared to other methods for detecting various steel surface defects, while also maintaining a relatively fast detection speed.

[0131] Table 2. Model Comparison Evaluation Indicators

[0132] method mAP50% FPS RPL 11.7 18.1 Unbiased-teacher 23.7 30.0 Softteacher 67 101.1 This method 68.7 57.8

[0133] Table 2 shows that this method exhibits superior performance, with the mAP value significantly higher than the previous method, reaching 85.7%. The FPS value is used to assess the smoothness of the image during steel inspection; a higher FPS indicates smoother movement and more information contained in the image. In the table, this method achieves an FPS value of 57.8, effectively capturing surface defects in the steel image. These data demonstrate the feasibility of the proposed solution and its improved prediction accuracy.

[0134] In summary, this embodiment uses NEV-CLS-64NEU-DETX and the NEU-CLS-64 dataset as the training dataset for the model. Based on the dataset, image preprocessing operations are performed, including mosaic data augmentation, adaptive anchor boxes, and adaptive image scaling to facilitate feature learning. The YOLOv5 model structure is used as the base model. To achieve better detection results, the original model is modified to improve performance. This includes replacing the 3×3 convolutional blocks of the ResNet152 network model with aggregation modules to enhance the network's feature extraction capabilities. This also allows for more effective capture of subtle defect features during the extraction of steel surface defects. Furthermore, this invention utilizes group spatial pyramid pooling in the feature learning layer to improve the original double pyramid structure, reducing the number of parameters through group convolution, thus making the module more efficient. Furthermore, the group convolution method, similar to regularization, can effectively reduce the risk of model fitting and improve the model's detection accuracy. In the prediction layer, relying on the feature map output by the feature learning layer, the RPN network is used to select the feature prediction region. To ensure the accuracy of the prediction, a parallel multi-path feature region selection method is adopted, calculating the error value between multiple selection boxes, and correcting the candidate boxes through regression variance, thereby improving the reliability of pseudo-label generation. This embodiment improves the feature extraction layer, feature learning layer, and prediction layer by modifying the YOLOv5 network model, enabling the model to accurately detect steel surface defects with limited data and efficiently detect the type and location of steel surface defects.

[0135] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0136] One or more embodiments of this specification are intended to cover all such alternative steel surface defect datasets falling within the broad scope of the appended claims; such datasets, modifications, and variations. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of this disclosure.

Claims

1. A semi-supervised image detection method for steel surface defects, characterized in that, include: S10. Obtain the steel image dataset; S20. Generate a training sample set based on the surface defect data in the steel image dataset; S30. Establish a steel surface defect detection model; The steel surface defect detection model is based on the YOLOv5 model, which includes a feature extraction layer, a feature learning layer, and a prediction layer. S31. The feature extraction layer uses an improved ResNet152 network model, and the specific steps include: S311. The steel surface defect image is convolved with a 1×1 kernel to obtain a first feature map; S312. Input the first feature map into the aggregation module to capture multi-scale features; The aggregation module downsamples and independently convolves the first feature map to generate multi-scale features; it samples the multi-scale features until they have the same resolution as the first feature map, and then connects them in the channel dimension to obtain the second feature map. S313. After the second feature map is processed by a 1×1 convolution kernel, it is residually connected with the first feature map to output the third feature map; S32. The third feature map is learned using a multi-scale feature learning structure; S33. The prediction layer predicts the types of defects on the iron surface and the candidate regions for defects on the steel surface, respectively; Among them, the candidate regions for steel table prediction are selected according to the multi-path parallel selection mechanism of the RPN network; S40. Train the steel surface defect detection model based on the training sample set, and obtain the steel surface defect detection model when the loss function value reaches the minimum. S50. Surface defect detection is performed on the steel image using the steel surface defect detection model; S60. Output the detection results of steel surface defect types.

2. The method of claim 1, wherein, S10 includes: The steel image dataset was obtained through datasets NEU-DET and NEU-CLS-64.

3. The method of claim 1, wherein, S20 includes: S21. The images in the training set are preprocessed; S22. Define the bounding box of the surface defect region in the preprocessed image; S23. The training sample set includes a labeled training sample set X = (x c ,y c ): 1 < c < N and unlabeled training sample set U = u c :1 < c < M, where c is a natural number; Where, x c Let y represent the c-th steel surface defect image in the labeled training sample set. c This indicates the surface defect category corresponding to the c-th steel surface defect image, u c Let N represent the c-th steel surface defect image in the unlabeled training sample set, and let N and M represent the number of steel surface defect images in the labeled training sample set and the unlabeled training sample set, respectively, with N>M.

4. The method of claim 3, wherein, S22 includes: The steel surface defect region box is the specific region of the steel surface defect. Each region box represents a steel surface defect region, denoted as A = [x, y, w, h]. Where x is the horizontal coordinate of the upper left corner of the steel surface defect area frame, y is the vertical coordinate of the steel surface defect area frame, is the width of the steel surface defect area frame, and h is the height of the steel surface defect area frame.

5. The method of claim 1, wherein, S32 includes: The multi-scale feature learning structure adopts group spatial pyramid pooling to improve the spatial pyramid pooling of the original dual feature pyramid structure. S321. The third feature map input yields three feature maps of different scales; S322. The three feature maps of different scales are grouped and convolved through the group spatial pyramid pooling to obtain grouped convolutional feature maps; S323. Perform multiple max pooling operations of different sizes on the group convolutional feature map, and then concatenate it with the group convolutional feature map; S324. Perform group convolution on the concatenated feature maps to obtain group spatial pooling feature maps; S325. Select the group spatial pooling feature map with the smallest scale and perform upsampling and downsampling operations, and perform residual connection with the three feature maps of different scales; S326. Output the fourth feature map.

6. The method of claim 5, wherein, The group space pyramid pooling includes: S322 to S324 include: a convolutional group layer with 32 1x1 convolutional kernels, 3 max pooling layers of different sizes, and a fully connected layer.

7. The method of claim 5, wherein, S33 includes: S331. Generate candidate regions for steel surface defect feature maps from the fourth feature map using RPN; S332. Extract candidate regions of the iron surface defect feature map from the region of interest to obtain the predicted feature map; S333. The predicted feature maps are formed by full connection to predict the types of defects on the steel surface and the regions of defects on the steel surface, respectively. The formula for calculating the defect area on the steel surface is: G x = P w d x (P) + P x + τ G y = P h d y (P) + P y + τ G w = P w exp(d w (P))+τ G h = P h exp(d h (P))+τ Where, P x P y P w and P h These are the x-coordinate, y-coordinate, height, and width of the center of the candidate region of the steel surface defect feature map, respectively; G x G y G h and G w Let x, y, height, and width be the center of the defect region on the steel surface. The candidate regions of the steel surface defect feature map are selected using multi-path parallel selection to determine the predicted feature map, and the selection error is calculated using the following formula: Wherein, τ is the average error value of the candidate region of the steel surface defect feature map; μ k It is the derived value of the k-th candidate region of the steel surface defect feature map extracted by the dithering frame algorithm of the RPN network; It is μ k The normalized value, h(b) k ) and w(b k ) represent the height and width of the candidate region for the k-th steel surface defect feature map, respectively.

8. The method according to claim 3, characterized in that, S40 includes: S41. Use the labeled training sample set to train and initialize the steel surface defect detection model; S42. Input the unlabeled training sample dataset into the initialized steel surface defect detection model, make predictions, and label it with pseudo-labels; S43. Train the initial steel surface defect detection model together with the pseudo-labeled training sample dataset and the labeled training sample set, optimize the parameters, and obtain the steel surface defect detection model.

9. The method of claim 1, wherein, The S60 includes: Output the detection results of steel surface defects; The detection results include the type of the steel surface defect and the target bounding box.

10. The method of claim 1, wherein, S40 includes: The loss function includes the loss for both labeled training samples and unlabeled training samples; the calculation formula is: L = L1 + L2 Where L1 is the loss function for training on labeled training sample sets, and L2 is the loss function for training on unlabeled training sample sets. The loss function for the labeled training sample set includes the cross-entropy loss function, which calculates the prediction loss and the error between the labeled defect location and the predicted defect location. The calculation formula is as follows: Where N represents the number of samples in the labeled training sample set, H() represents the cross-entropy, and P M () represents the probability distribution of the steel detection model, and τ1 represents the difference between the calibrated defect location and the predicted defect location. The loss function for unlabeled training samples is the cross-entropy loss between the predicted pseudo-label and the predicted label in the next round, and includes the error between the predicted target box and the labeled box. The calculation formula is as follows: Where M represents the number of samples in the unlabeled training sample set, H() represents the cross-entropy, and P M () represents the probability distribution of the steel detection model, and τ2 represents the difference between the calibrated defect location and the predicted defect location.

Images