Hot-rolled strip surface defect detection method based on double detection head D-yolo, storage medium and equipment

Abstract

The application discloses a hot-rolled strip steel surface defect detection method based on a double-detection-head D-YOLO, a storage medium and equipment, collects a hot-rolled strip steel surface defect image, carries out data enhancement processing, and obtains an enhanced defect data set; a detection layer of a YOLOv5 network is modified, a CA module and a Swin Transformer module are fused, and a double-detection-head D-YOLO convolutional neural network model is constructed by adopting an adaptive spatial feature fusion mode; the enhanced defect data set is input into the double-detection-head D-YOLO convolutional neural network model for training; a hot-rolled strip steel surface image is collected in real time, input into the trained double-detection-head D-YOLO convolutional neural network model, and a hot-rolled strip steel surface defect detection result is obtained. The application improves the inference speed while improving the hot-rolled strip steel surface defect detection precision.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, specifically relating to a method, storage medium, and equipment for detecting surface defects in hot-rolled strip steel based on the dual-detection-head D-YOLO. Background Technology

[0002] Hot-rolled strip steel is a type of steel product that has undergone hot rolling. It is typically produced by a hot-rolling strip mill during the hot rolling process, possessing high strength and toughness. After preheating, it enters the hot rolling mill for rolling. Hot-rolled strip steel is widely used in the manufacture of automotive structural components, machinery, shipbuilding, and building structures. During the production process, surface defects such as pitting, inclusions, and scratches are inevitable. These defects directly or indirectly affect the quality and performance of the strip steel. Therefore, defect detection of the produced strip steel is crucial for ensuring product quality and is essential for guaranteeing building safety.

[0003] Traditional methods for detecting surface defects in hot-rolled strip steel include visual inspection, magnetic particle testing, eddy current testing, and ultrasonic testing. However, these manual inspection methods are not only inefficient but also prone to missed or false detections. With the development of computer hardware, more and more researchers are exploring the application of machine vision algorithms in industrial surface defect detection. Generally, commonly used traditional machine learning algorithms for industrial defect detection include support vector machines, random forests, and Naive Bayes classifiers. Before using these algorithms, their performance was limited by feature representation, which required manual design. In recent years, with the rapid development of deep learning technology, object detection algorithms based on convolutional neural networks (CNNs) have been applied to detect industrial defect images, to some extent replacing traditional machine learning and manual feature design.

[0004] Recent research applied to hot-rolled strip steel defect detection largely focuses on either detection accuracy or speed. Models requiring higher accuracy typically demand more computational resources, including CPUs, GPUs, or other dedicated hardware. In industrial real-time inspection environments, limited computational resources hinder timely model inference. Models meeting real-time requirements often employ lightweight structures to reduce computational burden and improve inference speed. However, lightweight models are generally less capable of detecting complex scenes and small targets, resulting in relatively lower detection accuracy. Nevertheless, in industrial defect detection, while product quality is crucial, production volume cannot be reduced. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a method, storage medium, and equipment for detecting surface defects in hot-rolled strip steel based on a dual-detection-head D-YOLO, which improves the accuracy of surface defect detection in hot-rolled strip steel while increasing the inference speed.

[0006] To achieve the above technical objectives, the present invention adopts the following technical solution: a method for detecting surface defects in hot-rolled strip steel based on a dual-detection head D-YOLO, specifically including the following steps:

[0007] Step 1: Collect surface defect images of hot-rolled strip steel to obtain a defect image set. Perform data augmentation processing on each image in the defect image set to obtain an augmented defect dataset.

[0008] Step 2: Based on the YOLOv5 network, modify the detection layer of the YOLOv5 network and integrate the attention mechanism CA module and the Swing Transformer module to construct a dual-detection-head D-YOLO convolutional neural network model using an adaptive spatial feature fusion method.

[0009] Step 3: Input the enhanced defect dataset into the dual-detection-head D-YOLO convolutional neural network model for training until the CIOU loss function converges, thus completing the training of the dual-detection-head D-YOLO convolutional neural network model;

[0010] Step 4: Real-time acquisition of hot-rolled strip surface images, input into the trained dual-detection-head D-YOLO convolutional neural network model, to obtain the surface defect detection results of hot-rolled strip.

[0011] Furthermore, the data augmentation process in step 1 includes: changing the HSV of the hot-rolled strip image, image translation, image flipping, and mosaic processing.

[0012] Furthermore, the construction process of the dual-detection-head D-YOLO convolutional neural network model in step 2 is as follows: Based on the YOLOv5 network, the large target detection layer in the YOLOv5 network is removed, the shallowest feature extraction module C3 in the backbone network of the YOLOv5 network is modified to the feature extraction module CAC3 that integrates the attention mechanism CA, the feature fusion module for the detection layer in the neck network of the YOLOv5 network is modified to the feature fusion module CSTR that integrates the Swing Transformer module, and the prediction layer of the YOLOv5 network uses an adaptive spatial feature fusion method to detect the target box.

[0013] Furthermore, the specific process by which the feature extraction module CAC3, which integrates the attention mechanism CA, captures and extracts shallow features from the surface defect image of hot-rolled strip steel is as follows:

[0014] Step A.1: Take the output of the previous convolutional layer of the feature extraction module CAC3 as the input of the feature extraction module CAC3, and perform average pooling operations in both the height and width directions on the input feature map to obtain the feature maps in both the height and width directions respectively.

[0015] Step A.2: The feature maps in the height and width directions are stitched together. Through dimensionality reduction of convolutional layers, batch normalization and activation function processing, the position of each pixel in the hot-rolled strip surface defect image and the feature mapping map between the surrounding pixels are obtained.

[0016] Step A.3: Convolve the feature map according to the height and width of the original hot-rolled strip surface defect image to obtain feature maps with the same number of channels as the original. Combine the attention weights of the feature map on the height and width to obtain the shallow features on the hot-rolled strip surface defect image.

[0017] Furthermore, the network structure of the Swin Transformer module includes a window-type multi-head self-attention mechanism (W-MSA) layer, a first multilayer perceptron (MLP), a moving window-type multi-head self-attention mechanism (SW-MSA), and a second multilayer perceptron (MLP) connected in sequence. Normalization operations are performed before each of the W-MSA layer, the first MLP, the SW-MSA, and the second MLP, and residual connections are performed between each normalization layer to extract image features of surface defects in hot-rolled strip steel containing semantic features.

[0018] Furthermore, the process of using an adaptive spatial feature fusion method for target box detection is as follows:

[0019]

[0020] in, This represents the output vector of the pixel at row i and column j of the k-th level feature map, where k is selected from the feature map output by the CSTR feature extraction module fused with the SwinTransformer module or the feature map of the output layer of the neck network. This represents the feature vector at row i and column j after adjusting the feature map from level n to level k. express The corresponding weight parameters, express The corresponding weight parameters.

[0021] Furthermore, in the dual-head D-YOLO convolutional neural network model that removes the large object detection layer in the YOLOv5 network, each detection head predicts three detection boxes of different sizes. For each detection box, the confidence score of each detection box is calculated first, and detection boxes with confidence scores below the threshold are filtered out. For the remaining detection boxes, the NMS algorithm is used to eliminate detection boxes with large overlaps. The detection box with the highest confidence score is selected from the remaining detection boxes as the final detection box.

[0022] Furthermore, the CIOU loss function in step 3 is specifically as follows:

[0023]

[0024] Where w and h are the width and height of the predicted detection box, respectively. gt and h gt These are the width and height of the actual detection boxes, respectively. IOU is the ratio of the overlap area between the predicted and actual detection boxes to the area of ​​the actual detection boxes. A and B represent the actual and predicted detection boxes, respectively. ctr B ctr Let ρ represent the center points of A and B respectively. 2 (A ctr B ctr ) represents the Euclidean distance between the center points of the actual detection box A and the predicted detection box B, and C represents the diagonal distance of the smallest box that can enclose both boxes A and B.

[0025] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program that causes a computer to execute the described method for detecting surface defects in hot-rolled strip steel based on dual-head D-YOLO.

[0026] Furthermore, the present invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the aforementioned method for detecting surface defects in hot-rolled strip steel based on dual-head D-YOLO.

[0027] Compared with existing technologies, this invention has the following advantages: The hot-rolled strip steel surface defect detection method based on dual-head D-YOLO reduces the number of network layers by removing the large target detection layer in the YOLOv5 network, resulting in a dual-head D-YOLO network. This reduces the number of model parameters and computational load, allowing the model to focus more on learning the features of small targets, thereby improving the performance and accuracy of small target detection. Furthermore, by employing the CAC3 feature extraction module with a fusion attention mechanism (CA) in the shallowest feature extraction module, this invention better captures defect information from hot-rolled strip steel images with minimal model complexity. By using the CSTR feature fusion module with a fusion of Swing Transformer in the small target detection layer feature fusion module, this invention enhances the semantic information and feature representation of small-sized defects in hot-rolled strip steel images with minimal computational cost, strengthening the network's local perception capability. Finally, by establishing a spatial information interaction method between the detection head and the neck network output layer, each detection head can fuse more feature information from the hot-rolled strip steel image, thereby improving the overall detection effect. Compared with currently popular algorithms such as YOLOv3, YOLOv8, Faster R-CNN, Mask R-CNN, and SSD, it has significant advantages in both detection accuracy and speed for hot-rolled strip steel, making it more suitable for industrial applications. Attached Figure Description

[0028] Figure 1 This is a schematic flowchart of the hot-rolled strip surface defect detection method based on the dual-head D-YOLO of the present invention;

[0029] Figure 2 This is a schematic diagram of the dual-detector D-YOLO convolutional neural network model in this invention;

[0030] Figure 3 This is a schematic diagram of the SwinTransformer module in this invention;

[0031] Figure 4 This diagram illustrates a comparison of detection results using the detection method of this invention and the detection method based on the YOLOv5 network. Detailed Implementation

[0032] The technical solution of the present invention will be further explained and described below with reference to the accompanying drawings.

[0033] like Figure 1 This is a flowchart illustrating the surface defect detection method for hot-rolled strip steel based on the dual-head D-YOLO of the present invention. The method specifically includes the following steps:

[0034] Step 1: Collect surface defect images of hot-rolled strip steel to obtain a defect image set. Perform data augmentation processing on each image in the defect image set to obtain an enhanced defect dataset. The data augmentation processing in this invention includes: changing the HSV of the hot-rolled strip steel image, image translation, image flipping, and mosaic processing.

[0035] Step 2, as follows Figure 2To reduce costs and model complexity, and improve detection speed, a dual-detection-head D-YOLO convolutional neural network model was constructed based on the YOLOv5 network. The detection layers of the YOLOv5 network were modified, and the attention mechanism CA module and the Swing Transformer module were integrated. An adaptive spatial feature fusion method was used to construct the model. Specifically, the large target detection layer in the YOLOv5 network increases the number of parameters and computational cost. Since most defects in hot-rolled strip surface defect images are small, the large target detection layer in the YOLOv5 network was removed to reduce the number of parameters and computational cost, making the model more lightweight, improving inference speed, and allowing the model to focus more on learning the features of small targets, thereby improving the performance and accuracy of small target detection. The backbone network is responsible for feature extraction from the hot-rolled strip surface defect images, capturing semantic and structural information. It consists of convolutional layers, pooling layers, and activation layers, effectively converting the input hot-rolled strip surface defect images into high-level feature representations. To improve the performance of the dual-detection-head D-YOLO convolutional neural network model... To improve the performance of the backbone network, the shallowest feature extraction module C3 in the YOLOv5 backbone network was modified to a feature extraction module CAC3 with an attention mechanism CA. This explicitly models the interdependence between convolutional feature channels and positional information. This feature extraction mechanism enables the model to more effectively capture shallow features of hot-rolled strip surface defect images, thereby further enhancing detection performance. The neck network, located between the backbone network and the detection head, is responsible for further processing and fusing the features extracted by the backbone network to improve the expressive power and semantic information of the features. It consists of convolutional layers, upsampling operations, and cross-layer connections to achieve feature refinement and fusion. As the network deepens and multiple convolutional operations occur, feature information of small-sized defects in hot-rolled strip surface defect images is lost in higher-level feature maps. Therefore, the feature fusion module for the detection layer in the neck network of the YOLOv5 network was modified to use a fusion module with a Win The feature fusion module CSTR of the Transformer module can enhance the semantic information and feature representation of small-sized defects in hot-rolled steel strip surface defect images, thereby improving the detection accuracy of small-scale defects. By using an adaptive spatial feature fusion method to perform target box detection on the output layer of the YOLOv5 network that fuses the feature extraction module CSTR of the Swing Transformer module and the neck network, more feature information from hot-rolled steel strip surface defect images can be fused, effectively suppressing conflicting information between features at different levels and further improving detection accuracy.

[0036] The specific process by which the feature extraction module CAC3, which integrates the attention mechanism CA, is used in this invention to capture and extract shallow features on the surface defect image of hot-rolled strip steel is as follows:

[0037] Step A.1: Take the output of the convolution layer of the feature extraction module CAC3 as the input of the feature extraction module CAC3, and perform average pooling operations in the height and width directions on the input feature map to obtain the feature maps in the height and width directions respectively. This captures the long-term dependency along one spatial direction and saves the precise position information along the other spatial direction. This helps the network to more accurately locate the target of interest on the surface defect image of hot-rolled strip steel.

[0038] Step A.2: The feature maps in the height and width directions are stitched together. Through dimensionality reduction of convolutional layers, batch normalization and activation function processing, the position of each pixel in the hot-rolled strip surface defect image and the feature mapping map between the surrounding pixels are obtained.

[0039] Step A.3: Convolve the feature map according to the height and width of the original hot-rolled strip surface defect image to obtain feature maps with the same number of channels as the original. Combine the attention weights of the feature map on the height and width to obtain the shallow features on the hot-rolled strip surface defect image.

[0040] like Figure 3 In this invention, the network structure of the Swin Transformer module includes a window-type multi-head self-attention mechanism (W-MSA) layer, a first multilayer perceptron (MLP), a moving window-type multi-head self-attention mechanism (SW-MSA), and a second multilayer perceptron (MLP) connected in sequence. Normalization operations are performed before each of the W-MSA layer, the first MLP, the SW-MSA, and the second MLP, and residual connections are performed between each normalization layer to extract image features of surface defects in hot-rolled strip steel containing semantic features. The W-MSA layer can reduce the computational load of surface defect image features in hot-rolled strip steel.

[0041] The process of target bounding box detection using adaptive spatial feature fusion in this invention is as follows:

[0042]

[0043] in, This represents the output vector of the pixel at row i and column j of the k-th level feature map, where k is selected from the feature map output by the CSTR feature extraction module fused with the SwinTransformer module or the feature map of the output layer of the neck network. This represents the feature vector at row i and column j after adjusting the feature map from level n to level k. express The corresponding weight parameters, express The corresponding weight parameters.

[0044] In the dual-head D-YOLO convolutional neural network model that removes the large object detection layer in the YOLOv5 network, each detection head predicts three detection boxes of different sizes. For each detection box, the confidence score of each detection box is calculated first. Detection boxes with confidence scores below a threshold are filtered out. For the remaining detection boxes, the NMS algorithm is used to eliminate detection boxes with large overlaps. The detection box with the highest confidence score is selected as the final detection box from the remaining detection boxes.

[0045] Step 3: Input the enhanced defect dataset into the dual-detection-head D-YOLO convolutional neural network model for training until the CIOU loss function converges, thus completing the training of the dual-detection-head D-YOLO convolutional neural network model;

[0046] The CIOU loss function in this invention is specifically as follows:

[0047]

[0048] Where w and h are the width and height of the predicted detection box, respectively. gt and h gt These are the width and height of the actual detection boxes, respectively. IOU is the ratio of the overlap area between the predicted and actual detection boxes to the area of ​​the actual detection boxes. A and B represent the actual and predicted detection boxes, respectively. ctr B ctr Let ρ represent the center points of A and B respectively. 2 (A ctr B ctr ) represents the Euclidean distance between the center points of the actual detection box A and the predicted detection box B, and C represents the diagonal distance of the smallest box that can enclose both boxes A and B.

[0049] Step 4: Real-time acquisition of hot-rolled strip surface images, input into the trained dual-detection-head D-YOLO convolutional neural network model to obtain the surface defect detection results of hot-rolled strip, and mark the defect-free hot-rolled strip as qualified products.

[0050] The surface defect detection method for hot-rolled strip steel based on dual-head D-YOLO of this invention was compared with the surface defect detection results of hot-rolled strip steel using the YOLOv5 network. The detection accuracy of the dual-head D-YOLO network model designed in this invention is 0.989 in the mean accuracy (mAP)0.5, while the detection accuracy of the YOLOv5 network is 0.862. The detection method of this invention improves the accuracy by 12.7% compared with the detection method of YOLOv5. Figure 4 This invention improves the accuracy of images of various defects, cracks, inclusions, patches, pits, indented oxide scale, and scratches on the surface of hot-rolled strip steel, and can meet the requirements of real-time detection in industry.

[0051] In one embodiment of the present invention, a computer-readable storage medium is also provided, storing a computer program that enables a computer to execute the hot-rolled strip surface defect detection method based on dual-head D-YOLO.

[0052] In one technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the hot-rolled strip steel surface defect detection method based on dual-detection head D-YOLO.

[0053] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for detecting surface defects in hot-rolled strip steel based on a dual-head D-YOLO detector, characterized in that, Specifically, the steps include the following: Step 1: Collect surface defect images of hot-rolled strip steel to obtain a defect image set. Perform data augmentation processing on each image in the defect image set to obtain an augmented defect dataset. Step 2: Based on the YOLOv5 network, modify the detection layer of the YOLOv5 network, and integrate the attention mechanism CA module and the Swin Transformer module to construct a dual-detection-head D-YOLO convolutional neural network model using an adaptive spatial feature fusion method. The construction process of the dual-detection-head D-YOLO convolutional neural network model is as follows: Based on the YOLOv5 network, the large target detection layer in the YOLOv5 network is removed, the shallowest feature extraction module C3 in the backbone network of the YOLOv5 network is modified to the feature extraction module CAC3 that integrates the attention mechanism CA, the feature fusion module for the detection layer in the neck network of the YOLOv5 network is modified to the feature fusion module CSTR that integrates the Swing Transformer module, and the prediction layer of the YOLOv5 network uses an adaptive spatial feature fusion method to detect the target box for the feature extraction module CSTR that integrates the Swing Transformer module and the output layer of the neck network. The specific process by which the feature extraction module CAC3 of the fusion attention mechanism CA captures and extracts shallow features on the surface defect image of hot-rolled strip steel is as follows: Step A.1: Take the output of the previous convolutional layer of the feature extraction module CAC3 as the input of the feature extraction module CAC3, and perform average pooling operations in both the height and width directions on the input feature map to obtain the feature maps in both the height and width directions respectively. Step A.2: The feature maps in the height and width directions are stitched together. Through dimensionality reduction of convolutional layers, batch normalization and activation function processing, the position of each pixel in the hot-rolled strip surface defect image and the feature mapping map between the surrounding pixels are obtained. Step A.3: Convolve the feature map according to the height and width of the original hot-rolled strip surface defect image to obtain feature maps with the same number of channels as the original. Combine the attention weights of the feature map on the height and width to obtain the shallow features on the hot-rolled strip surface defect image. The network structure of the Swin Transformer module includes a window-type multi-head self-attention mechanism (W-MSA) layer, a first multilayer perceptron (MLP), a moving window-type multi-head self-attention mechanism (SW-MSA), and a second multilayer perceptron (MLP) connected in sequence. Normalization operations are performed before each of the W-MSA layer, the first MLP, the SW-MSA, and the second MLP, and residual connections are performed between each normalization layer to extract the surface defect image features of hot-rolled strip steel containing semantic features. The process of target bounding box detection using an adaptive spatial feature fusion method is as follows: in, Indicates the first Level feature map in OK The output vector of the pixel at the column position. Feature maps selected from the CSTR output of the feature extraction module fused with the Swin Transformer module or feature maps from the output layer of the neck network. The feature map represents the first... Level to the After level adjustment OK eigenvectors at column positions express The corresponding weight parameters, express The corresponding weight parameters; Step 3: Input the enhanced defect dataset into the dual-detection-head D-YOLO convolutional neural network model for training until the CIOU loss function converges, thus completing the training of the dual-detection-head D-YOLO convolutional neural network model; Step 4: Real-time acquisition of hot-rolled strip surface images, input into the trained dual-detection-head D-YOLO convolutional neural network model, to obtain the surface defect detection results of hot-rolled strip.

2. The method for detecting surface defects in hot-rolled strip steel based on dual-head D-YOLO as described in claim 1, characterized in that, The data augmentation process in step 1 includes: changing the HSV of the hot-rolled strip image, image translation, image flipping, and mosaic processing.

3. The method for detecting surface defects in hot-rolled strip steel based on dual-head D-YOLO as described in claim 1, characterized in that, In the dual-head D-YOLO convolutional neural network model that removes the large object detection layer in the YOLOv5 network, each detection head predicts three detection boxes of different sizes. For each detection box, the confidence score of each detection box is calculated first. Detection boxes with confidence scores below a threshold are filtered out. For the remaining detection boxes, the NMS algorithm is used to eliminate detection boxes with large overlaps. The detection box with the highest confidence score is selected as the final detection box from the remaining detection boxes.

4. The method for detecting surface defects in hot-rolled strip steel based on dual-head D-YOLO as described in claim 1, characterized in that, The CIOU loss function in step 3 is as follows: in, and These are the width and height of the predicted detection box, respectively. and These are the width and height of the actual detection box, respectively. It is the ratio of the overlap area between the predicted and ground truth bounding boxes to the area of ​​the ground truth bounding boxes. , Representing the actual detection boxes and the predicted detection boxes, , They represent and The center point, This represents the Euclidean distance between the center points of the actual detection box A and the predicted detection box B. Indicates the ability to surround , The diagonal distance between the smallest of the two boxes.

5. A computer-readable storage medium storing a computer program, characterized in that, The computer program causes the computer to execute the hot-rolled strip surface defect detection method based on the dual-head D-YOLO as described in any one of claims 1-4.

6. An electronic device, characterized in that, include: The method for detecting surface defects in hot-rolled strip steel based on dual-head D-YOLO as described in any one of claims 1-4 includes a memory, a processor, and a computer program stored in the memory and executable on the processor.

Images