Metal Surface Defect Detection Method and Device Based on Improved YOLO Model

Abstract

This application relates to a metal surface defect detection method and device based on an improved YOLO model. The method includes: obtaining a metal surface defect data set. Constructing a metal surface defect detection model. Downsampling a first feature map through a reparameterized DSC structure in the first group of feature layer modules to obtain first feature information, and the first feature information is downsampled through a reparameterized CSP structure in the first group of feature layer modules and input into the neck part for feature fusion to obtain first fusion information. This process continues until the remaining three groups of feature layer modules complete the iterative sampling and aggregation of the metal surface defect image samples between different feature layers, obtaining the fusion information corresponding to each group of feature layer modules. The fusion information extracts fusion feature maps of different scales through the neck part, and then through the processing of the metal surface defect detection model, a sample detection result is obtained. Using this method can improve the real-time detection accuracy of small-size metal defect images.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular, to a metal surface defect detection method and device based on an improved YOLO model. Background Art

[0002] Metal products are prone to defects such as surface oxidation, fracture, depression, and scratches during the manufacturing process. These defects will have an adverse impact on the quality and durability of the products, resulting in potential economic losses. Therefore, metal surface defect detection plays a crucial role in ensuring the quality of industrial products. Since the 1980s, researchers have adopted vision-based defect detection methods. Typical traditional detection algorithms include image preprocessing, ROI (region of interest) detection, image segmentation, feature extraction, and defect classification. Traditional algorithms achieve defect classification and detection by manually extracting features. However, the features extracted manually cannot fully express the information in complex environments, and the defect performance is greatly affected by the manually extracted features. Problems such as large computational amounts, defect positions, and sizes are still difficult to solve.

[0003] Since 2012, convolutional neural networks have become increasingly important in computer vision tasks. Two-stage deep learning algorithms represented by Faster-RCNN aim at defect classification and location regression. Using the region proposal network (RPN), they predict region images to generate candidate boxes, and finally extract features that fuse ROI (region of interest). However, the algorithm has a large memory occupancy, computational redundancy, and poor computational effects. Compared with two-stage algorithms, one-stage object detection algorithms such as YOLO and SSD use regression techniques to combine classification and localization tasks, directly calculating the classification results and position coordinates of multiple targets. The algorithm has a lower complexity and faster detection time. Although convolutional neural networks (CNNs) have strong feature extraction capabilities, as the depth of the model increases, it becomes more difficult to extract features from smaller targets. In addition, due to background interference and the diverse shapes of defects, CNNs are hindered in accurately detecting and classifying surface defects. Summary of the Invention

[0004] Based on this, in order to solve the above technical problems, it is necessary to provide a metal surface defect detection method and device based on an improved YOLO model that can perform high-precision detection on small-sized metal defect images.

[0005] A metal surface defect detection method based on an improved YOLO model, the method includes:

[0006] Obtain a metal surface defect data set. The metal surface defect data set contains multiple metal surface defect image samples with different sizes and resolutions. The metal surface defect image samples are labeled by defect type.

[0007] Construct a metal surface defect detection model. The metal surface defect detection model uses an improved YOLO model as the basic framework. The backbone part of the metal surface defect detection model includes a first feature map extraction module and four groups of feature layer modules connected in sequence. Each group of feature layer modules includes a reparameterized DSC structure and a reparameterized CSP structure.

[0008] The first feature map is downsampled by the reparameterized DSC structure in the first group of feature layer modules to obtain first feature information. The first feature information is downsampled by the reparameterized CSP structure in the first group of feature layer modules and input to the neck part for feature fusion to obtain first fusion information.

[0009] The first fusion information is input to the second group of feature layer modules for feature sampling aggregation until the remaining three groups of feature layer modules complete the iterative sampling aggregation of the metal surface defect image samples between different feature layers, obtaining the fusion information corresponding to the remaining three groups of feature layer modules.

[0010] The fusion information extracts fusion feature maps of different scales through the neck part, and the fusion feature maps are detected and processed by the detection head part of the metal surface defect detection model to obtain the detection results of the metal surface defect image samples.

[0011] In one embodiment, the metal surface defect dataset is divided into a training set of metal surface defect data, a validation set of metal surface defect data, and a test set of metal surface defect data according to a preset ratio, and the training set of metal surface defect data is selected as the metal surface defect image samples.

[0012] In one embodiment, the first feature map acquisition module includes: a CBS convolution module and a preprocessing module. The CBS convolution module filters redundant information of the feature map of the metal surface defect image sample to obtain the first feature map to be processed. The first feature map to be processed is sequentially reparameterized by the preprocessing module to obtain the first feature map.

[0013] In one embodiment, it further includes: performing repeated convolution operations on the first feature map by the reparameterized DSC structure in the first group of feature layer modules to capture the first feature information of the first feature map. The first feature information is downsampled by the reparameterized CSP structure in the first group of feature layer modules and input to the context transformer module in the neck part to extract the context information of the first feature information, and the feature weights of the first feature information are adjusted according to the context information of the first feature information to obtain the first fusion information.

[0014] In one embodiment, it further includes: inputting the first fusion information into the reparameterized DSC structure of the second group of feature layer modules to perform repeated convolution operations on the first fusion information to capture the second feature information of the first fusion information. The second feature information is downsampled by the reparameterized CSP structure in the second group of feature layer modules and input into the context transformer module in the neck part to extract the context information of the second feature information. The feature weights of the second feature information are adjusted according to the context information of the second feature information to obtain the second fusion information. Inputting the second fusion information into the reparameterized DSC structure of the third group of feature layer modules to perform repeated convolution operations on the second fusion information to capture the third feature information of the second fusion information. The third feature information is downsampled by the reparameterized CSP structure in the third group of feature layer modules and input into the context transformer module in the neck part to extract the context information of the third feature information. The feature weights of the third feature information are adjusted according to the context information of the third feature information to obtain the third fusion information. Inputting the third fusion information into the reparameterized DSC structure of the fourth group of feature layer modules to perform repeated convolution operations on the third fusion information to capture the fourth feature information of the third fusion information. The fourth feature information is downsampled by the reparameterized CSP structure in the fourth group of feature layer modules and input into the spatial pyramid pooling module for pooling operations to obtain the fourth pooled feature information. The fourth pooled feature information is input into the context transformer module in the neck part to extract the context information of the fourth pooled feature information. The feature weights of the fourth pooled feature information are adjusted according to the context information of the fourth pooled feature information to obtain the fourth fusion information.

[0015] In one embodiment, the neck part adopts a general feature pyramid network model. The general feature pyramid network model includes several reparameterized DSC structures, several reparameterized CSP structures, several feature tensor connection points, and 3 extended sampling scales.

[0016] In one embodiment, it further includes: the first fusion information, the second fusion information, the third fusion information, and the fourth fusion information respectively extract the fusion features of different scales between the feature layers through the general feature pyramid network model in the neck part to obtain the first fusion feature map corresponding to the first fusion information, the second fusion feature map corresponding to the second fusion information, the third fusion feature map corresponding to the third fusion information, and the fourth fusion feature map corresponding to the fourth fusion information. The first fusion feature map, the second fusion feature map, the third fusion feature map, and the fourth fusion feature map are respectively detected and processed by the detection head part of the metal surface defect detection model, and the detection results of 4 metal surface defect image samples are obtained in order from large to small according to the scale.

[0017] In one embodiment, it further includes: performing image enhancement training on the metal surface defect detection model according to the defect type of the metal surface defect image sample and the detection result obtained by inputting the metal surface defect image sample into the metal surface defect detection model to obtain a trained metal surface defect detection model. Inputting the metal surface defect image to be detected into the trained metal surface defect detection model to obtain the detection image of the metal surface defect image to be detected.

[0018] In one embodiment, the image enhancement training includes image flipping, mosaic, and HSV conversion.

[0019] A metal surface defect detection device based on an improved YOLO model, the device includes:

[0020] A dataset acquisition module for acquiring a metal surface defect dataset. The metal surface defect dataset contains multiple metal surface defect image samples with different sizes and resolutions. The metal surface defect image samples are labeled by defect type.

[0021] A model construction module for constructing a metal surface defect detection model. The metal surface defect detection model uses an improved YOLO model as the basic framework. The backbone part of the metal surface defect detection model includes a first feature map extraction module and four groups of feature layer modules connected in sequence. Each group of feature layer modules includes a reparameterized DSC structure and a reparameterized CSP structure.

[0022] A first fusion information acquisition module for downsampling the first feature map through the reparameterized DSC structure in the first group of feature layer modules to obtain first feature information, and downsampling the first feature information through the reparameterized CSP structure in the first group of feature layer modules and inputting it into the neck part for feature fusion to obtain first fusion information.

[0023] A fusion information acquisition module for inputting the first fusion information into the second group of feature layer modules for feature sampling aggregation until the remaining three groups of feature layer modules complete the iterative sampling aggregation of the metal surface defect image sample between different feature layers to obtain the fusion information corresponding to the remaining three groups of feature layer modules.

[0024] A detection result module for the fusion information to extract fusion feature maps of different scales through the neck part, and the fusion feature maps are detected and processed by the detection head part of the metal surface defect detection model to obtain the detection result of the metal surface defect image sample.

[0025] The above metal surface defect detection method and device based on the improved YOLO model obtain a data set containing metal surface defect image samples of multiple different sizes and resolutions. Each sample is labeled with a tag of the defect type. Based on the improved YOLO model as the basic framework, a metal surface defect detection model is constructed. Among them, the backbone part of the model is sequentially connected by a first feature map extraction module and four groups of feature layer modules. And each group of feature layer modules includes a reparameterized DSC structure and a CSP structure, which can extract richer and high-resolution feature information, which helps to enhance the model's perception and localization ability of small-size defects. Furthermore, using structural reparameterization, the first feature map is upsampled by the reparameterized DSC structure in the first group of feature layer modules to obtain the first feature information. By upsampling and downsampling operations, the feature information of the same layer and adjacent layers is fused, thereby increasing the non-linear representation ability of the network and improving the accuracy and scale diversity of feature extraction. Further, the first feature information is downsampled by the reparameterized CSP structure in the first group of feature layer modules and input into the neck part for feature fusion to obtain the first fusion information. Then, the first fusion information is input into the second group of feature layer modules for feature sampling aggregation. Sequentially complete the iterative sampling aggregation of the metal surface defect image samples by the remaining three groups of feature layer modules between different feature layers to obtain the fusion information corresponding to the remaining three groups of feature layer modules. Cross-scale connections are used to connect the neck part and the backbone part, which includes two paths: one path is the skip connection of the same level, and the other path is the cross-scale connection that fuses high-level semantic information and low-level geometric information. Through the iterative sampling aggregation between feature layer modules, information transmission and fusion between different feature layers are realized to capture the feature representations of defects at different scales and comprehensively utilize multi-scale features to improve the detection accuracy. Finally, the fusion information extracts fusion feature maps of different scales through the neck part, and these fusion feature maps are detected and processed by the detection head part of the metal surface defect detection model, and finally the detection results of the metal surface defect image samples are generated, thus realizing high-precision metal surface defect detection. Description of the Drawings

[0026] Figure 1 It is an application scenario diagram of the metal surface defect detection method based on the improved YOLO model in an embodiment;

[0027] Figure 2 It is a schematic flowchart of the metal surface defect detection method based on the improved YOLO model in an embodiment;

[0028] Figure 3 It is a structural block diagram of CoT-GFPN in the metal surface defect detection model in an embodiment;

[0029] Figure 4Schematic diagram of a defect detection data set in an embodiment;

[0030] Figure 5 Schematic diagram of the structure of a metal surface defect detection device based on an improved YOLO model in an embodiment. Detailed implementation manners

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] The metal surface defect detection method provided by the present application can be applied to, for example, Figure 1 the metal surface defect detection model shown. Among them, the metal surface defect detection model uses an improved YOLOv5 model as the basic framework, and its model architecture includes a backbone part (Backbone), a neck part (Neck network), and a detection head part (Head). Among them, the backbone part includes a CBS convolution module, 5 feature layers (P2 - P6), and a spatial pyramid pooling module (Spatial Pyramid Pooling Feature, hereinafter referred to as SPPF). The neck part includes four contextual transformer modules (Contextual Transformer, hereinafter referred to as CoT), several feature tensor connection points (concat), 4 up - sampling scales (UP - Sample), several re - parameterized CSP structures (Rep - Cross Stage Partial, hereinafter referred to as Rep - CSP), and several re - parameterized DSC structures (Rep - Dynamic Scaling Convolution, hereinafter referred to as Rep - DSC). The detection head part includes 4 convolution modules (Conv). In addition, the 5 feature layers are connected in sequence from top to bottom. Starting from the second feature layer (P3) to the fifth feature layer (P6), each feature layer is connected to the neck part through the CoT module for image data connection.

[0033] In one embodiment, as Figure 2 shown, a metal surface defect detection method based on an improved YOLO model is provided. Taking this method applied to Figure 1 the metal surface defect detection model as an example, it includes the following steps:

[0034] Step 202, obtain a metal surface defect data set.

[0035] The metal surface defect dataset contains multiple metal surface defect image samples with different sizes and resolutions. The metal surface defect image samples are labeled by defect types, where the defect types on the metal surface can be cracks, inclusions, patches, surface pitting, rolled-in scale, and scratches. Additionally, according to a preset ratio, the metal surface defect map dataset corresponding to each defect type is divided into data types, and each defect type forms a corresponding dataset according to the same ratio. Among them, 80%, 10%, and 10% of the metal surface defect map dataset are respectively divided into the training set, the validation set, and the test set. The training set is taken as the sample dataset of the metal surface defect images and is input into the metal surface defect detection model.

[0036] Specifically, the metal surface defect dataset to be detected, NEU-DET, is obtained through the CoT module in the backbone part of the metal surface defect detection model. The image scale corresponding to this dataset is 640*640*3 (i.e., H×W×C: H is the height, W is the width, and C is the number of channels), and it contains 1,800 images. Each image is a grayscale image of 200*200 pixels, among which there are 300 images for each defect type.

[0037] Step 204, construct a metal surface defect detection model.

[0038] The metal surface defect detection model uses the improved YOLOv5 model as the basic framework. The backbone part of the original YOLOv5 model uses the CSPDarkNet53 structure. In this structure, the Rep-CSP structure is introduced to replace the original bottleneck CSP structure (i.e., BottleneckCSP), and the Rep-DSC structure is added as the convolutional module to form a new backbone part, so that the new backbone part contains 5 downsamplings and 5 residual blocks. Each downsampling and residual block constitutes a set of feature layers, and 5 sets of feature layer modules (P2, P3, P4, P5, P6) are connected after the CoT module and are connected end to end in sequence from top to bottom. Each set of feature layer modules includes at least 1 reparameterized DSC structure and 1 reparameterized CSP structure. Among them, the first feature layer module P2 is a feature map preprocessing module. The data detection set processed by the CoT module passes through the Rep-DSC structure of the current feature layer, and the depthwise separable convolution of the data detection is reparameterized. The 3×3 depth convolution and the 1×1 point convolution are reparameterized, and the result of fusing Conv (convolution operation) and BN (batch normalization) is input into the Rep-CSP structure of the current layer:

[0039] Conv(X) = X′ = w k *X

[0040]

[0041]

[0042] Among them, X is the data detection set, that is, the metal surface defect detection sample, and w k is the convolution weight coefficient, the average value of batch normalization is μ, the standard deviation of batch normalization is σ, the weight coefficient w b of batch normalization, the bias of batch normalization is b, and a minimum parameter ε of batch normalization to prevent the denominator from dropping to 0. In addition, the added non-linear residual block of this structure includes a Bottleneck layer to adjust the number of channels, thereby obtaining the first feature map. In addition, each group of feature layer modules includes 1 Rep-DSC structure and 1 Rep-CSP structure.

[0043] Step 206, downsample the first feature map through the reparameterized DSC structure in the first group of feature layer modules to obtain the first feature information. The first feature information is downsampled through the reparameterized CSP structure in the first group of feature layer modules and input to the neck part for feature fusion to obtain the first fusion information.

[0044] Specifically, the first feature map is downsampled through the Rep-DSC structure of the P3 of the first group of feature layer modules. The standard 3×3 convolution is decoupled into a multi-branch structure composed of non-convolution, 1×1 convolution, and 3×3 convolution. Then, the size of each branch convolution kernel is converted into the first feature information of 3×3. This information is input into the CoT module in the neck part. Using the multi-head attention mechanism MSA of this module, context encoding is performed on the picture information matrix key (K) composed of all adjacent matrices in each convolution kernel to obtain the static context information matrix K1 of the first feature information.

[0045] Furthermore, the static context information key K1 of the current convolution kernel is concatenated with the query matrix Q of the Queen-fusion input picture feature information, and then the attention matrix A is obtained through two consecutive 1×1 convolutions. After combining the attention matrix A with the value matrix V with feature information obtained through one 1×1 convolution, the dynamic context information matrix K2 can be obtained. Furthermore, the static context information matrix K1 and the dynamic context information matrix K2 are fused for feature fusion to obtain the first fusion information.

[0046] Step 208, input the first fusion information into the second group of feature layer modules for feature sampling aggregation until the remaining three groups of feature layer modules complete the iterative sampling aggregation of the metal surface defect image samples between different feature layers, and obtain the fusion information corresponding to the remaining three groups of feature layer modules.

[0047] The first fusion information is input into the Rep-DSC structure of the feature layer module P4 in the second group to perform repeated convolution operations on the first fusion information to capture the second feature information of the first fusion information. The second feature information is downsampled by the Rep-CSP structure in the feature layer module of the second group and input into the context transformer module in the neck part to extract the context information of the second feature information. The feature weights of the second feature information are adjusted according to the context information of the second feature information to obtain the second fusion information. The second fusion information is input into the Rep-DSC structure of the feature layer module P5 in the third group to perform repeated convolution operations on the second fusion information to capture the third feature information of the second fusion information. The third feature information is downsampled by the Rep-CSP structure in the feature layer module of the third group and input into the context transformer module in the neck part to extract the context information of the third feature information. The feature weights of the third feature information are adjusted according to the context information of the third feature information to obtain the third fusion information. The third fusion information is input into the Rep-DSC structure of the feature layer module P6 in the fourth group to perform repeated convolution operations on the third fusion information to capture the fourth feature information of the third fusion information. The fourth feature information is downsampled by the Rep-CSP structure in the feature layer module of the fourth group and input into the spatial pyramid pooling module for pooling operations to obtain the fourth pooled feature information. The fourth pooled feature information is input into the context transformer module in the neck part to extract the context information of the fourth pooled feature information. The feature weights of the fourth pooled feature information are adjusted according to the context information of the fourth pooled feature information to obtain the fourth fusion information.

[0048] Step 210, the fusion information extracts fusion feature maps of different scales through the neck part, and the fusion feature maps are detected and processed by the detection head part of the metal surface defect detection model to obtain the detection results of the metal surface defect image samples.

[0049] Specifically, the first fusion information, the second fusion information, the third fusion information, and the fourth fusion information respectively extract fusion features of different scales between feature layers through the Generalized Feature Pyramid Network model (hereinafter referred to as GFPN) of the Neck network, and the four groups of fusion information are input into the Neck network. The Neck network uses cross-scale connections, which include two paths: one path is the skip connection of the same level, and the other path is the cross-scale connection Queen-fusion (abbreviated as Q) that fuses high-level semantic information and low-level geometric information.

[0050] Further, each feature layer module from P3 to P6 in the backbone part forms a CoT-GFPN model with the GFPN of the Neck network through the CoT module, and performs upsampling and downsampling operations on the fused information after reparameterizing each layer, so as to fuse the feature information of the same layer and adjacent layers, and obtain the first fused feature map corresponding to the first fused information, the second fused feature map corresponding to the second fused information, the third fused feature map corresponding to the third fused information, and the fourth fused feature map corresponding to the fourth fused information.

[0051] Further, the first fused feature map, the second fused feature map, the third fused feature map, and the fourth fused feature map are respectively subjected to detection processing by the Head of the metal surface defect detection model, and the detection results of 4 metal surface defect image samples are obtained in order from large to small according to the scale.

[0052] Further, the metal surface defect detection model is subjected to image enhancement training according to the defect type of the metal surface defect image sample and the detection result obtained by inputting the metal surface defect image sample into the metal surface defect detection model, and the trained metal surface defect detection model is obtained. In addition, the enhancement training can be image flipping, Mosaic, and HSV conversion.

[0053] Further, the metal surface defect image to be detected is input into the trained metal surface defect detection model to obtain the detection image of the metal surface defect image to be detected.

[0054] The above metal surface defect detection method based on the improved YOLO model obtains a dataset containing metal surface defect image samples of multiple different sizes and resolutions. Each sample is labeled with a tag of the defect type. Based on the improved YOLO model as the basic framework, a metal surface defect detection model is constructed. Among them, the backbone part of the model is sequentially connected by a first feature map extraction module and four groups of feature layer modules. And each group of feature layer modules includes repeated DSC structures and CSP structures, which can extract richer and high-resolution feature information, which helps to enhance the model's perception and localization ability for small-size defects. Furthermore, using structural reparameterization, the first feature map is upsampled through the reparameterized DSC structure in the first group of feature layer modules to obtain the first feature information. By upsampling and downsampling operations, the feature information of the same layer and adjacent layers is fused, thereby increasing the non-linear representation ability of the network and improving the accuracy and scale diversity of feature extraction. Further, the first feature information is downsampled through the reparameterized CSP structure in the first group of feature layer modules and input into the neck part for feature fusion to obtain the first fusion information. Then, the first fusion information is input into the second group of feature layer modules for feature sampling aggregation. Sequentially complete the iterative sampling aggregation of the metal surface defect image samples by the remaining three groups of feature layer modules between different feature layers to obtain the fusion information corresponding to the remaining three groups of feature layer modules. The neck part is connected to the backbone part using cross-scale connections, which includes two paths: one path is the skip connection of the same level, and the other path is the cross-scale connection that fuses high-level semantic information and low-level geometric information. Through the iterative sampling aggregation between feature layer modules, information transfer and fusion between different feature layers are realized to capture the feature representations of defects at different scales and comprehensively utilize multi-scale features to improve the detection accuracy. Finally, the fusion information extracts fusion feature maps of different scales through the neck part, and these fusion feature maps are detected and processed through the detection head part of the metal surface defect detection model, and finally the detection results of the metal surface defect image samples are generated, thus realizing high-precision metal surface defect detection.

[0055] In one embodiment, the metal surface defect dataset is divided into a training set of metal surface defect data, a validation set of metal surface defect data, and a test set of metal surface defect data according to a preset ratio, and the training set of metal surface defect data is selected as the metal surface defect image samples.

[0056] In one embodiment, the first feature map acquisition module includes: a CBS convolution module and a preprocessing module. The CBS convolution module filters the redundant information of the feature map of the metal surface defect image sample to obtain the first feature map to be processed. The first feature map to be processed is sequentially reparameterized by the preprocessing module to obtain the first feature map.

[0057] It should be noted that the first feature map acquisition module consists of two parts: the CBS convolution module and the preprocessing module. First, the CBS convolution module is used to screen the redundant information of the feature map of the metal surface defect image samples, which helps to remove redundant features and extract more useful feature information. Then, the preprocessing module performs reparameterization processing on the first feature map to be processed, further optimizing the representation ability of the feature map, enhancing its sensitivity and discrimination ability to metal surface defects, and thus obtaining the first feature map with more details.

[0058] In one embodiment, the first feature information of the first feature map is captured by performing repeated convolution operations on the first feature map through the reparameterized DSC structure in the first group of feature layer modules. The first feature information is downsampled by the reparameterized CSP structure in the first group of feature layer modules and input into the context transformer module in the neck part to extract the context information of the first feature information. The feature weights of the first feature information are adjusted according to the context information of the first feature information to obtain the first fusion information.

[0059] It should be noted that by performing repeated convolution operations, the useful feature information in the first feature map is extracted, which helps to better extract and represent metal surface defects. Furthermore, through the downsampling operation of the reparameterized CSP structure, the size of the feature map is reduced and feature fusion is achieved, combining the first feature information with other feature information. Finally, through the context transformer module, the context information of the first feature information is extracted, and these information can help to better understand the semantics and context relationship of the defects. Thus, according to these context information, the feature weights of the first feature information are adjusted, further optimizing its representation ability, improving the performance and accuracy of the metal surface defect detection model, and enabling it to better adapt to different types and sizes of defect samples.

[0060] In one embodiment, the first fusion information is input into the reparameterized DSC structure of the feature layer module of the second group to perform repeated convolution operations on the first fusion information to capture the second feature information of the first fusion information. The second feature information is downsampled by the reparameterized CSP structure in the feature layer module of the second group and input into the context transformer module in the neck part to extract the context information of the second feature information. The feature weights of the second feature information are adjusted according to the context information of the second feature information to obtain the second fusion information. The second fusion information is input into the reparameterized DSC structure of the feature layer module of the third group to perform repeated convolution operations on the second fusion information to capture the third feature information of the second fusion information. The third feature information is downsampled by the reparameterized CSP structure in the feature layer module of the third group and input into the context transformer module in the neck part to extract the context information of the third feature information. The feature weights of the third feature information are adjusted according to the context information of the third feature information to obtain the third fusion information. The third fusion information is input into the reparameterized DSC structure of the feature layer module of the fourth group to perform repeated convolution operations on the third fusion information to capture the fourth feature information of the third fusion information. The fourth feature information is downsampled by the reparameterized CSP structure in the feature layer module of the fourth group and input into the spatial pyramid pooling module for pooling operations to obtain the fourth pooled feature information. The fourth pooled feature information is input into the context transformer module in the neck part to extract the context information of the fourth pooled feature information. The feature weights of the fourth pooled feature information are adjusted according to the context information of the fourth pooled feature information to obtain the fourth fusion information.

[0061] In one embodiment, the neck part adopts a general feature pyramid network model. The general feature pyramid network model includes a number of reparameterized DSC structures, a number of reparameterized CSP structures, a number of feature tensor connection points, and 4 extended sampling scales.

[0062] In one embodiment, the first fusion information, the second fusion information, the third fusion information, and the fourth fusion information respectively extract the fusion features of different scales between the feature layers through the general feature pyramid network model in the neck part, and obtain the first fusion feature map corresponding to the first fusion information, the second fusion feature map corresponding to the second fusion information, the third fusion feature map corresponding to the third fusion information, and the fourth fusion feature map corresponding to the fourth fusion information. The first fusion feature map, the second fusion feature map, the third fusion feature map, and the fourth fusion feature map are respectively detected and processed by the detection head part of the metal surface defect detection model, and the detection results of 4 metal surface defect image samples are obtained in order from large to small according to the scale.

[0063] It should be noted that through the layer-by-layer processing and feature fusion of the first fusion information, the second fusion information, the third fusion information, and the fourth fusion information, a multi-scale fusion feature map is obtained. These fusion feature maps are detected through the detection head of the metal surface defect detection model, realizing the detection of metal surface defect image samples. Specifically, through repeated convolution operations and downsampling, each feature layer module extracts features from the fusion information of the previous layer, and obtains context information and adjusts feature weights through the context transformer module. With the gradual stacking of feature layer modules, the feature information is further extracted and adjusted.

[0064] In addition, the neck part adopts a general feature pyramid network model, which includes multiple reparameterized DSC structures, reparameterized CSP structures, feature tensor connection points, and extended sampling scales, enabling the model to retain and utilize multi-scale feature information and adapt to defects of different scales. Finally, the fusion feature maps are detected through the detection head of the metal surface defect detection model, and the detection results of metal surface defect image samples are obtained scale by scale from large to small, thereby improving the accuracy and robustness of metal surface defect detection, especially having good adaptability to defect samples of different sizes and resolutions.

[0065] In one embodiment, the metal surface defect detection model is trained for image enhancement according to the defect type of the metal surface defect image sample and the detection result obtained by inputting the metal surface defect image sample into the metal surface defect detection model, obtaining a trained metal surface defect detection model. The metal surface defect image to be detected is input into the trained metal surface defect detection model to obtain the detection image of the metal surface defect image to be detected.

[0066] In one embodiment, the image enhancement training includes image flipping, mosaic, and HSV conversion.

[0067] In one of the embodiments, the YOLOv5 algorithm adopts the CSPDarkNet53 structure in the backbone Backbone network, which contains the BottleneckCSP structure with a residual network, realizing the effective extraction of feature maps. The initial parameters of the input image are 640×640×3 (H×W×C: H is the height, W is the width, and C is the number of channels), and image enhancement methods such as Mosaic data augmentation, adaptive anchor box calculation, image flipping, image scaling, and changing image size are adopted during the training dataset. In the Head output layer, three feature layers of 80×80×256, 40×40×512, and 20×20×1024 will be extracted. After using a 1×1 convolutional layer with a stride of 1 and a sigmoid activation function, the number of channels can be adjusted to 3×(N + 5). In the YOLO v5 algorithm, the initial image is divided into multiple grids, and each grid is responsible for detecting objects in the corresponding area. Three bounding boxes will be designed to output three types of parameters: 1) Four values corresponding to the bounding box, namely the center coordinates (x, y) of the box and the width and height (w, h) of the box; 2) Confidence, with a value range of 0 to 1; 3) N represents the number of categories in the dataset, corresponding to a set of conditional class detection probabilities for the categories on the dataset, with a value range of 0 to 1. The final output of the convolutional layer is a three-dimensional tensor. The YOLOv5 model is improved through the following steps:

[0068] Step 1: In the Backbone part, replace the original BottleneckCSP and ordinary convolution with the reparameterized CSP structure Rep-CSP and the reparameterized depthwise separable convolution Rep-DSC respectively, to reduce the number of parameters and increase the non-linear diversity connections in the network.

[0069] Step 2: At the four feature layers of 80×80, 40×40, 20×20, and 10×10 in the Backbone part, introduce the CoT module to enhance the feature extraction information ability of each feature layer and transmit the extracted information to the Neck network.

[0070] Step 3: Replace the FPN+PANet network in the Neck part with the CoT-GFPN network as shown in Figure 3 shown.

[0071] Step 4: Improve the Head part into four detection heads to generate four different sizes of detection boxes of 80×80, 40×40, 20×20, and 10×10. At the same time, the initial anchor boxes need to be regenerated using the k-means clustering algorithm in YOLO for the initial positioning of the target. As shown in Table 1:

[0072] Table 1 Initial Anchor Boxes and Feature Map Sizes of the Detection Layers

[0073]

[0074] Step Five: As Figure 4 shown, use the metal surface defect detection dataset NEU-DET to divide a total of 1800 images from six datasets into a training set, a validation set, and a test set, with a ratio of 8:1:1. Among them, the validation set proofreads the training results, and the test set records data such as the sampling speed and detection time during the training process.

[0075] Step Six: Train the dataset using image enhancement methods such as image flipping, Mosaic, and HSV conversion.

[0076] It should be understood that although Figures 1-3 the steps in the flowchart of Figures 1-3 are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover,

[0077] In one embodiment, as Figure 5 shown, a metal surface defect detection device based on an improved YOLO model is provided, including: a dataset acquisition module 502, a model construction module 504, a first fusion information acquisition module 506, a fusion information acquisition module 508, and a detection result module 510, where:

[0078] The dataset acquisition module 502 is used to obtain the metal surface defect dataset. The metal surface defect dataset contains multiple metal surface defect image samples with different sizes and resolutions. The metal surface defect image samples are labeled by defect type.

[0079] The model construction module 504 is used to construct a metal surface defect detection model. The metal surface defect detection model uses an improved YOLO model as the basic framework. The backbone part of the metal surface defect detection model includes a first feature map extraction module and four groups of feature layer modules connected in sequence. Each group of feature layer modules includes a reparameterized DSC structure and a reparameterized CSP structure.

[0080] The first fusion information acquisition module 506 is configured to upsample the first feature map through the reparameterized DSC structure in the first group of feature layer modules to obtain first feature information. The first feature information is downsampled through the reparameterized CSP structure in the first group of feature layer modules and input to the neck part for feature fusion to obtain first fusion information.

[0081] The fusion information acquisition module 508 is configured to input the first fusion information into the second group of feature layer modules for feature sampling and aggregation until the remaining three groups of feature layer modules complete the iterative sampling and aggregation of the metal surface defect image samples between different feature layers, obtaining the fusion information corresponding to the remaining three groups of feature layer modules.

[0082] The detection result module 510 is configured to extract fusion feature maps of different scales from the fusion information through the neck part, and the fusion feature maps are detected and processed by the detection head part of the metal surface defect detection model to obtain the detection result of the metal surface defect image sample.

[0083] For the specific limitations of the metal surface defect detection device based on the improved YOLO model, reference can be made to the limitations of the metal surface defect detection method based on the improved YOLO model in the above text, which will not be elaborated here. Each module in the above metal surface defect detection device based on the improved YOLO model can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above respective modules.

[0084] Those skilled in the art can understand that Figure 5 the structure shown in

[0085] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memories can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0086] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0087] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for detecting metal surface defects based on an improved YOLO model, characterized in that, The method includes: Obtaining a metal surface defect dataset; the metal surface defect dataset contains multiple metal surface defect image samples with different sizes and resolutions; the metal surface defect image samples are labeled by defect type; Constructing a metal surface defect detection model; the metal surface defect detection model uses an improved YOLO model as the basic framework; the backbone part of the metal surface defect detection model includes a first feature map extraction module and four groups of feature layer modules connected in sequence; each group of the feature layer modules includes a reparameterized DSC structure and a reparameterized CSP structure; Downsampling the first feature map through the reparameterized DSC structure in the first group of feature layer modules to obtain first feature information, and downsampling the first feature information through the reparameterized CSP structure in the first group of feature layer modules and inputting it into the neck part for feature fusion to obtain first fusion information; Inputting the first fusion information into the second group of feature layer modules for feature sampling aggregation until the remaining three groups of feature layer modules complete the iterative sampling aggregation of the metal surface defect image samples between different feature layers to obtain the fusion information corresponding to the remaining three groups of feature layer modules; The fusion information extracts fusion feature maps of different scales through the neck part, and the fusion feature maps are detected and processed by the detection head part of the metal surface defect detection model to obtain the detection results of the metal surface defect image samples.

2. The method according to claim 1, wherein The metal surface defect dataset is divided into a training set, a validation set, and a test set of the metal surface defect data according to a preset ratio, and the training set of the metal surface defect data is selected as the metal surface defect image samples.

3. The method according to claim 1, wherein The first feature map acquisition module includes: a CBS convolution module and a preprocessing module; Filtering redundant information of the feature map of the metal surface defect image sample through the CBS convolution module to obtain a first feature map to be processed; The first feature map to be processed is sequentially reparameterized through the preprocessing module to obtain a first feature map.

4. The method according to claim 1, wherein Downsampling the first feature map through the reparameterized DSC structure in the first group of feature layer modules to obtain first feature information, and downsampling the first feature information through the reparameterized CSP structure in the first group of feature layer modules and inputting it into the neck part for feature fusion to obtain first fusion information, including: Performing repeated convolution operations on the first feature map through the reparameterized DSC structure in the first group of feature layer modules to capture the first feature information of the first feature map, downsampling the first feature information through the reparameterized CSP structure in the first group of feature layer modules and inputting it into the context transformer module of the neck part to extract the context information of the first feature information, and adjusting the feature weight of the first feature information according to the context information of the first feature information to obtain first fusion information.

5. The method according to claim 4, wherein Input the first fusion information into the feature layer module of the second group for feature sampling and aggregation, until the remaining three groups of feature layer modules complete the iterative sampling and aggregation of the metal surface defect image sample between different feature layers, obtaining the fusion information corresponding to the remaining three groups of feature layer modules, including: Input the first fusion information into the reparameterized DSC structure of the feature layer module of the second group to perform repeated convolution operations on the first fusion information to capture the second feature information of the first fusion information. The second feature information is downsampled by the reparameterized CSP structure in the feature layer module of the second group and input into the context transformer module in the neck part to extract the context information of the second feature information. Adjust the feature weights of the second feature information according to the context information of the second feature information to obtain the second fusion information; Input the second fusion information into the reparameterized DSC structure of the feature layer module of the third group to perform repeated convolution operations on the second fusion information to capture the third feature information of the second fusion information. The third feature information is downsampled by the reparameterized CSP structure in the feature layer module of the third group and input into the context transformer module in the neck part to extract the context information of the third feature information. Adjust the feature weights of the third feature information according to the context information of the third feature information to obtain the third fusion information; Input the third fusion information into the reparameterized DSC structure of the feature layer module of the fourth group to perform repeated convolution operations on the third fusion information to capture the fourth feature information of the third fusion information. The fourth feature information is downsampled by the reparameterized CSP structure in the feature layer module of the fourth group and input into the spatial pyramid pooling module for pooling operations to obtain the fourth pooled feature information. The fourth pooled feature information is input into the context transformer module in the neck part to extract the context information of the fourth pooled feature information. Adjust the feature weights of the fourth pooled feature information according to the context information of the fourth pooled feature information to obtain the fourth fusion information.

6. The method according to claim 5, characterized in that The neck part adopts a general feature pyramid network model; the general feature pyramid network model includes several reparameterized DSC structures, several reparameterized CSP structures, several feature tensor connection points, and 4 extended sampling scales.

7. The method according to claim 6, characterized in that, The fusion information extracts fusion feature maps of different scales through the neck part, and the fusion feature maps are detected and processed by the detection head part of the metal surface defect detection model to obtain the detection result of the metal surface defect image sample, including: The first fusion information, the second fusion information, the third fusion information, and the fourth fusion information respectively extract fusion features of different scales between feature layers through the general feature pyramid network model of the neck part, and obtain a first fusion feature map corresponding to the first fusion information, a second fusion feature map corresponding to the second fusion information, a third fusion feature map corresponding to the third fusion information, and a fourth fusion feature map corresponding to the fourth fusion information; The first fusion feature map, the second fusion feature map, the third fusion feature map, and the fourth fusion feature map are respectively detected and processed by the detection head part of the metal surface defect detection model, and detection results of 4 metal surface defect image samples are obtained in order from largest to smallest scale.

8. The method according to claim 1, characterized in that, After the fusion information extracts fusion feature maps of different scales through the neck part, and the fusion feature maps are detected and processed by the detection head part of the metal surface defect detection model to obtain the detection results of the metal surface defect image samples, it further includes: Performing image enhancement training on the metal surface defect detection model according to the defect type of the metal surface defect image sample and the detection result obtained by inputting the metal surface defect image sample into the metal surface defect detection model, to obtain the trained metal surface defect detection model; Inputting the metal surface defect image to be detected into the trained metal surface defect detection model to obtain the detection image of the metal surface defect image to be detected.

9. The method according to claim 8, wherein The image enhancement training includes image flipping, mosaic, and HSV conversion.

10. A metal surface defect detection device based on an improved YOLO model, characterized in that, The device includes: A dataset acquisition module, configured to acquire a metal surface defect dataset; the metal surface defect dataset contains multiple metal surface defect image samples of different sizes and resolutions; the metal surface defect image samples are labeled by defect type; A model construction module, configured to construct a metal surface defect detection model; the metal surface defect detection model uses an improved YOLO model as the basic framework; the backbone part of the metal surface defect detection model includes a first feature map extraction module and four groups of feature layer modules connected in sequence; each group of the feature layer modules includes a reparameterized DSC structure and a reparameterized CSP structure; A first fusion information acquisition module, configured to downsample the first feature map through the reparameterized DSC structure in the first group of feature layer modules to obtain first feature information, and the first feature information is downsampled through the reparameterized CSP structure in the first group of feature layer modules and input to the neck part for feature fusion to obtain first fusion information; A fusion information acquisition module, configured to input the first fusion information into the second group of feature layer modules for feature sampling aggregation, until the remaining three groups of feature layer modules complete iterative sampling aggregation of the metal surface defect image sample between different feature layers, to obtain the fusion information corresponding to the remaining three groups of feature layer modules; The detection result module is used to extract fusion feature maps of different scales from the fusion information through the neck part, and the fusion feature maps are processed by the detection head part of the metal surface defect detection model to obtain the detection results of the metal surface defect image samples.

Images