Metal surface defect detection method

Abstract

The invention discloses a metal surface defect detection method. The method comprises the following steps: establishing a metal surface defect image data set; the metal surface defect image is preprocessed, and it is guaranteed that multiple defect types exist in the same image; a YOLOv7 network model is improved; carrying out training evaluation on the metal surface defect data set by using the improved YOLOv7 model, and selecting proper hyper-parameters in the training process; data obtained after model training are analyzed and judged, and the model is adjusted and optimized; and keeping the optimal model weight, and carrying out experimental testing on the trained model. The method comprises the following steps: establishing a multi-type metal surface defect data set; a polarization self-attention mechanism and an SPDConv module are introduced into the neck network, and important information is captured in a self-adaptive mode while image information is reserved; and an NWD loss function is introduced, and the positioning capability of the network on small-size defects is improved. The method can meet the requirements of metal surface defect detection under different working conditions, and has good robustness.

Description

Technical Field

[0001] The present invention relates to a machine vision and target detection method, and in particular to a metal surface defect detection method based on improved YOLOv7. Background Art

[0002] During the production and processing of metal parts, various defects such as cracks, scratches, inclusions, etc. often appear on the surface due to the influence of various complex factors. The surface quality of metal parts will directly affect the operating efficiency and life of the product, and is more likely to affect the operating safety and stability of the equipment. Therefore, metal surface defect detection is very important. Traditional methods commonly used for steel plate surface defect detection include manual detection, photoelectric detection, and infrared detection. The manual detection method uses manual visual inspection and simple touch inspection, which is highly subjective and inefficient. The photoelectric detection method consumes more resources, is difficult to detect minor scratches, and is prone to misjudgment or missed detection when the defect types are similar or complex. The infrared detection method can detect fewer types of defects and has poor versatility.

[0003] The target detection algorithms used in traditional machine vision detection are all based on feature extraction based on candidate regions, which has high time complexity and long model training time. In addition, due to the diversity of the morphology of the detected targets, the robustness of the image features is poor. Deep learning technology has become a research hotspot in machine vision detection with its strong versatility. For defect data sets of different objects, it can achieve corresponding intelligent surface defect detection after adaptive training. The detection algorithms based on deep learning are mainly divided into two categories from the perspective of ideas: one is a two-step detection algorithm, which divides target detection into two parts: generating candidate boxes and identifying target categories. Typical representatives include Mask R-CNN, Faster R-CNN and other algorithms. The advantage is high detection accuracy. The other is a single-step detection algorithm, which directly predicts the category of the object from the picture. Typical representatives include YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector) and other algorithms. The advantage is fast detection speed. Patent 202410574924.6 uses the Faster R-CNN model to modify the loss function and add an attention mechanism, achieving high-precision detection of aluminum surface datasets and impeller surface datasets respectively. Patent 201910277616.6 uses an improved SSD model to detect images in the second network by learning the image information with annotated boxes in the first network, thus completing the target detection task.

[0004] Since it was proposed in 2015, the YOLO target detection series algorithm has continued to evolve and become one of the most outstanding target detectors in the single-stage target detection model. It converts the target detection problem into a logistic regression problem, supports multi-label objects and maintains accuracy while having a high detection speed. Chinese patent 202110017704.X is based on YOLOv3. By modifying the backbone network model, introducing a pyramid pooling network, and increasing the output of large-size feature maps, it realizes target detection in underwater images, and compares the Faster R-CNN, SSD and YOLOv2 algorithms to verify the superiority of the improved algorithm in detection accuracy. Chinese patent 202410163902.0 is based on the YOLOv7 model. By adding an attention mechanism and a multi-scale feature aggregation module and modifying the loss function, it realizes object detection in X-ray images. It compares the Faster R-CNN two-step detection algorithm with single-step detection algorithms such as SSD and YOLOv3, reflecting the advantages of the YOLO series of algorithms over other single-step and two-step algorithms, and proves that the improved YOLO algorithm improves indicators such as average accuracy of all categories and frames per second when processing target detection tasks. Chinese patent 202310678915.7 is based on YOLOX, replaces the backbone network, and performs multi-scale enhancement and refinement on the feature map to realize visual detection of metal surface defects, and uses a public data set for experimental verification.

[0005] The YOLO series of algorithms have made great progress in the field of target detection and are constantly updated. They are widely used in surface defect detection with their fast and accurate detection capabilities. However, for small-size target detection such as metal surface defects, due to the low pixel ratio, feature information is easily lost, resulting in low detection accuracy. In addition, there are certain deviations in detection accuracy in different data sets and different environments. Summary of the invention

[0006] Purpose of the invention: The purpose of the present invention is to provide a metal surface defect detection method to solve the problem that the types of metal surface defects are diverse and the detection accuracy of the same network for different data sets is different.

[0007] Technical solution: A metal surface defect detection method according to the present invention comprises the following steps:

[0008] Establish a metal surface defect image dataset;

[0009] Preprocess metal surface defect images to ensure that multiple defect types exist in the same image;

[0010] Improve the YOLOv7 network model;

[0011] Use the improved YOLOv7 model to train and evaluate the metal surface defect dataset, and select appropriate hyperparameters during the training process;

[0012] Analyze and judge the data after model training, and adjust and optimize the model;

[0013] The optimal model weights are retained and the trained model is experimentally tested.

[0014] Preferably, the establishing of the metal surface defect image dataset comprises:

[0015] By using the surface defect images of metal parts actually collected on the production line and combining them with public image datasets, we can realize the image collection of various types of metal surface defects in different environments; mark the location and category information of each surface defect in the image, expand the images in the dataset, and randomly divide them into training set, test set and validation set.

[0016] Furthermore, the image acquisition of various types of metal surface defects under different environments includes:

[0017] The public NUE-DET, GC10 steel surface image dataset, Tianchi aluminum profile defect dataset and the self-made dataset of metal parts surface defect images were used to screen the typical defects on the metal surface, and target detection was performed for 10 types of surface defects, including cracks, plaques, inclusions, indentations, punchings, oxidized surfaces, rolled-in oxide scales, scratches, stains and silk spots.

[0018] Furthermore, the marking of the position and category information of each surface defect in the image includes:

[0019] Use the LabelImg annotation tool to mark the location and category information of each metal surface defect in yolo format;

[0020] The randomly dividing the images in the data set into a training set, a test set and a validation set after expanding the images in the data set includes:

[0021] The images in the dataset are expanded by translation, mirroring and chromaticity changes, and randomly divided into training set, test set and validation set in the ratio of 8:1:1.

[0022] Preferably, the preprocessing of the metal surface defect image comprises:

[0023] The detection model loads the input metal surface image into memory, sets mosaic data enhancement, randomly crops, scales and reassembles four images to form a new image, so that there are multiple defect types in one image, and enhances the robustness parameters of the network for different surface defect detection; sets the input image size to be uniformly processed as 640×640; automatically verifies and modifies the initial anchor frame size parameters so that the best recall rate of the anchor frame on the metal surface defect dataset is greater than or equal to 0.98.

[0024] Preferably, the improved YOLOv7 network model includes:

[0025] The metal surface defect image is input into the backbone network for feature extraction, feature fusion is performed through the neck network, non-maximum suppression operation is performed through the head network, and the detection result is output; based on the CIOU loss function, the normalized Wasserstein distance loss function is introduced to improve the positioning accuracy of small-sized targets; the SPDConv module is added before the small-sized prediction head to improve the accuracy of small targets and obtain effective metal surface defect features; the polarization self-attention module is added before the downsampling operation of the neck network to enhance the network feature extraction capability.

[0026] Furthermore, based on the CIOU loss function, the normalized Wasserstein distance loss function is introduced, including:

[0027] Based on the CIOU loss function used in YOLOv7, the normalized Wasserstein distance loss function is introduced as a metric, that is, the rectangular box is modeled as a Gaussian distribution, and then the distribution distance is used to measure the similarity between the rectangular boxes; the Wasserstein distance can reflect W2 when the two distributions do not overlap or the overlapping part is negligible. 2 (N a ,N b ) distribution distance;

[0028] The normalized Wasserstein distance NWD is calculated as follows:

[0029]

[0030] C is a constant related to the data set, N a and N b They are respectively the real box A=(c xa ,c ya ,ω a ,h a ) and prediction box B = (c xb ,c yb ,ω b ,h b ) modeled Gaussian distribution, W2 2 (N a ,N b ) is the Wasserstein distance of the Gaussian distribution modeled for the annotation box;

[0031] The ratio between CIOU and NWD loss is adjusted to 5:5, and the sum of the two is used as the final loss function value to balance the requirements of defect detection tasks of different sizes.

[0032] Further, adding the SPDConv module before the small-size prediction head includes:

[0033] The SPDConv module consisting of a space-to-depth layer and a non-strided convolution layer is added before the small-size image prediction head to reduce image information loss; the feature extraction of the original input image is performed to obtain the feature Figure X , firstly, a series of sub-feature maps f are obtained through the SPD layer x，y , which is a downsampling operation; then the sub-feature maps are connected along the channel dimension to obtain the intermediate feature Figure X ';Finally, the non-strided convolutional layer is used to filter the final features Figure X ". The small size can be 20×20 size.

[0034] Further, the adding of a polarization self-attention module before the downsampling operation of the neck network includes:

[0035] Three parallel polarized self-attention mechanisms are added before the two downsampling of the YOLOv7 neck network to keep the network at a high resolution in the channel and spatial dimensions.

[0036] Another aspect of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0037] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0038] Firstly, in order to solve the problem of large size differences of metal surface defects, the present invention adds the NWD loss function on the basis of the original CIOU loss function to improve the network's adaptability to surface defects of different sizes.

[0039] Secondly, to address the difficulty in detecting low-resolution images and small-size defects, an SPDConv module is added before the 20×20 prediction head to improve detection accuracy;

[0040] Finally, in view of the diversity of defect types, the polarization self-attention module is integrated into the feature fusion network to strengthen the attention and utilization of important features and improve the network's feature extraction and generalization capabilities. A variety of metal surface defect image datasets are established, and the metal parts images collected on the production line are combined with public datasets to train the network for metal surface defect detection under different materials and different environments.

[0041] Through these improvements, the present invention improves the adaptability of the method to surface detection of various metal parts in different environments, making it more suitable for multi-type complex metal surface defect detection tasks and having better robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is the metal surface defect detection algorithm structure based on YOLOv7;

[0043] Figure 2 This is the existing YOLOv7 network model diagram;

[0044] Figure 3 SPDConv schematic diagram when scale = 2;

[0045] Figure 4 This is the structure diagram of the polarized self-attention mechanism;

[0046] Figure 5 This is the improved YOLOv7 network model diagram;

[0047] Figure 6 Ten types of metal surface defect images from the public dataset used;

[0048] Figure 7 A production line diagram where metal surface defect images are collected;

[0049] Figure 8 is the collected metal surface defect image;

[0050] Fig. 9 It is the F1 confidence curve;

[0051] Fig.10 It is the Loss curve graph. DETAILED DESCRIPTION

[0052] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0053] A metal surface defect detection method is as follows:

[0054] S1. Constructing a dataset: In order to achieve high-precision visual inspection of surface defects of metal parts with different shapes and sizes under different working conditions, a metal surface defect detection method based on improved YOLOv7 is invented. Therefore, it is necessary to first establish a dataset of multiple types of metal surface defect images. In the construction of this dataset, the defect images of metal parts collected on the production line are used, combined with three public datasets: NEU-DET, GC10 steel surface image dataset and Tianchi aluminum profile defect dataset. Ten common metal surface defects in manufacturing are selected as detection targets: cracks, plaques, inclusions, indentations, punching, oxidized surface, rolled oxide scale, scratches, stains, and silk spots. The target type images are selected, and the defect images of the same type in different datasets are merged to achieve image acquisition of multiple types of metal surface defects under different environments. Use the LabelImg annotation tool to annotate the location and category information of each surface defect in the yolo format. In order to balance the number of different defects in different datasets and fully show the characteristics of each defect and the differences between different defects, the images in the dataset are expanded by translation, mirroring and chromaticity changes, so that the number of images of each defect reaches more than 300. The final metal surface defect dataset contains 7266 images, which are randomly divided into training set, test set and validation set in a ratio of 8:1:1.

[0055] NEU-DET: For more information, see the paper by He et al., “An end-to-end steel surface defected detection approach via fusing multiple hierarchical features” (HE Y, SONG K, MENG Q, et al. An end-to-end steel surface defected detection approach via fusing multiple hierarchical features [J]. IEEE Transactions on instrumentation and measurement, 2019, 69(4): 1493-1504.), which can be downloaded from https: / / ieeexplore.ieee.org / abstract / document / 8709818 / .

[0056] GC10 steel surface image dataset: For detailed information, please refer to the paper "Deep metallic surface defect detection: The new benchmark and detection network" by LV et al. (LV XM, DUANF J, JIANG JJ, et al. Deep metallic surface defect detection: The new benchmark and detection network [J]. Sensors, 2020, 20 (6): 1562.), which can be downloaded from https: / / www.mdpi.com / 1424-8220 / 20 / 6 / 1562.

[0057] Tianchi aluminum profile defect dataset: For detailed information, please refer to the paper "Aluminum profile surface defect detection method based on machine vision" by Zhao Wenhong et al. (Zhao Wenhong, Zhou Shente, Lv Jianbiao, Zhang Xiao, Wang Yuyu. Aluminum profile surface defect detection method based on machine vision [J]. Journal of Zhejiang University of Technology, 2021, 49(01): 76-81.), which can be downloaded from https: / / kns.cnki.net / kcms2 / article / abstract? v=zLXkCTFmbAGvnkqy7wGBB30TMAKJF7zQiUZ6FrRXQvhq9P_F25mkQsjgkc6ycuKc5b6W4PxKvGcUNjLRgYLoFZBeKztjzUQv425Vl AR5qwPbg4m3nlBxGErBt1uZqzSODeF35IF9GykDrOj8IoBxxXMkx3UTLV90-OGoDFsW_pCRoVveoyWT5_EReP1qx8WP_5Q6yFpTaRg=&uniplatform=NZKPT&language=CHS Download.

[0058] S2. Image preprocessing: The detection model loads the input metal surface image into memory, sets mosaic data enhancement, randomly crops, scales, and reassembles four images to form a new image, so that there are multiple defect types in one image, and enhances the robustness parameters of the network for different surface defect detection; sets the input image size to 640×640; automatically verifies and modifies the initial anchor frame size parameters so that the best recall rate of the anchor frame on the metal surface defect dataset is greater than or equal to 0.98.

[0059] S3. Build an improved YOLOv7 network model: First, input the metal surface defect image into the backbone network for feature extraction. Then, perform feature fusion through the neck network, add three polarization self-attention modules before the two downsampling operations of the neck network to enhance the network feature extraction capability; add an SPDConv module before the 20×20 size prediction head to improve the accuracy of small targets and obtain effective metal surface defect features. Finally, perform non-maximum suppression operation of the head network, and introduce the NWD loss function based on the CIOU loss function in the operation to improve the positioning accuracy of small-sized targets. Take 50% of CIOU and NWD respectively, add the two as the final loss function value, and output the final detection result.

[0060] Method Model YOLOv7 detector: For detailed information, please refer to the paper "YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors" by Wang et al. (Wang CY, Bochkovskiy A, Liao HY M. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[J].arXiv preprint arXiv:2207.02696,2022.), which can be downloaded from https: / / openaccess.thecvf.com / content / CVPR2023 / html / Wang_YOLOv7_Trainable_Bag-of-Freebies_Sets_New_State-of-the-Art_for_Real-Time_Object_Detectors_CVPR_2023_paper.html).

[0061] In step S3, the YOLO network uses the intersection-over-union (IOU) ratio as the rectangular box similarity metric, which is calculated by dividing the intersection of the predicted box and the true box by the union of the two, that is:

[0062]

[0063] Among them, A is the area covered by the predicted box, and B is the area covered by the real box. IOU has the advantages of non-negativity and symmetry, but when the two rectangular boxes have no intersection, the similarity cannot be calculated. In addition, IOU cannot give the details of the intersection between the rectangular boxes, and cannot measure the similarity of the center point distance, length and width between the rectangular boxes, resulting in the IOU metric being not equally sensitive to objects of different scales, and having low performance for small-size object detection. YOLOv7 uses CIOU as a metric, and considers the overlapping area, center point distance and aspect ratio of the two rectangular boxes on the basis of IOU. The calculation method is:

[0064]

[0065] where b, b gt Represent the coordinates of the center points of the predicted box and the real box respectively, ρ represents the Euclidean distance between the two center points, and c represents the diagonal length of the minimum bounding rectangle that can contain both the predicted box and the real box. α is the weight function, and its calculation formula is:

[0066]

[0067] v is the similarity of the aspect ratio of the two rectangular boxes, and its calculation formula is:

[0068]

[0069] w, w gt Represent the width of the predicted box and the real box, h, h gt Represent the height of the predicted box and the true box respectively. The calculation formula of CIOU loss is:

[0070] CIOU loss =1-CIOU

[0071] The larger the CIoU value, the smaller the loss, which means that the predicted box output by the model is closer to the real box. CIOU takes into account the similarity of the center point distance, length and width between rectangular boxes on the basis of IOU, but is still very sensitive to small-sized target position deviations. For small-sized defects on metal surfaces such as inclusions, indentations, and contamination, the present invention uses the normalized Wasserstein distance (Normalized Wasserstein Distance, NWD) as a metric, that is, the rectangular box is modeled into a Gaussian distribution, and then the distribution distance is used to measure the similarity between the rectangular boxes. The Wasserstein distance can reflect W2 when there is no overlap between the two distributions or the overlapping part is negligible. 2 (N a ,N b ) distribution. Assume that the bounding box R = (c x ,cy ,w,h), where (c x ,c y ), w and h represent the center coordinates, width and height of the bounding box respectively. Then let the real box A = (c xa ,c ya ,w a ,h a ), prediction box B = (c xb ,c yb ,w,h). NWD is calculated as:

[0072]

[0073] C is a constant related to the data set, N a and N b The Gaussian distribution N(μ, ∑) that models the real box A and the predicted box B respectively obeys W2 2 (N a ,N b ) is the Gaussian distribution N modeling the annotation box a and N b The Wasserstein distance is calculated as:

[0074]

[0075] The NWD loss calculation formula is:

[0076] NWD loss =1-NWD(N a ,N b )

[0077] Adding NWD loss to the loss function can make up for the deficiency of IOU loss in small object detection. At the same time, the ratio between CIOU and NWD loss is adjusted to 5:5. The sum of the two is used as the final loss function value to balance the requirements of defect detection tasks of different sizes. The final loss value is:

[0078] Loss=CIOU loss ×0.5+NWD loss ×0.5

[0079] In step S3, the original YOLOv7 network uses more strided convolution and pooling layers for image classification and target detection tasks, which easily leads to the loss of fine-grained information during feature learning and is not effective in processing low-resolution images and small targets. In view of the problems of low resolution and small size of metal surface defect images, the present invention introduces the SPDConv module before the 20×20 size prediction head to improve the small target detection effect. For the original input image, the size of the feature (S, S, C1) is obtained after feature extraction. Figure X First, a series of sub-feature maps f with sizes (S / scale, S / scale, C1) are obtained through the SPD layer x，y , which is a downsampling operation. Then the sub-feature maps are connected along the channel dimension to obtain a size of (S / scale, S / scale, scale 2 C1) Intermediate features Figure X '. Finally, the non-strided convolutional layer is used to filter the final feature with a size of (S / scale, S / scale, C2) Figure X ”.

[0080] In the metal surface defect image detection task in step S3, the downsampling process needs to process the data at a low spatial resolution to obtain high-dimensional features, but this also leads to a large loss of image detail information and structural information. Therefore, the polarized self-attention (PSA) mechanism is introduced before the downsampling process to keep the network at a high resolution in the channel and spatial dimensions, and reduce the information loss caused by dimensionality reduction without increasing the number of parameters. In view of the differences in a variety of metal surface defect images in different environments, the polarized self-attention mechanism helps the module capture important contextual information in the image, improves sensitivity to the target, and focuses on the target area, reducing background interference, making the model more robust and effective when processing images with different backgrounds and multiple targets. Polarized self-attention (PSA) has two branches, one branch is a self-attention mechanism in the channel dimension, and the other branch is a self-attention mechanism in the spatial dimension. Finally, the results of these two branches are fused to obtain the output of the polarized self-attention structure.

[0081] S4. Model training: Use the improved YOLOv7 model to train and evaluate the metal surface defect dataset, and select appropriate hyperparameters during the training process.

[0082] S5. Experimental testing: Analyze and judge the data after model training, and adjust and optimize the model. Keep the optimal model weights and conduct experimental testing on the trained model.

[0083] Figure 1This is the structure of the metal surface defect detection algorithm based on YOLOv7. The YOLOv7 model is mainly composed of the backbone network (backbone), the neck network (neck) and the head network (head). The backbone network is responsible for feature extraction. The backbone network of YOLOv7 is an efficient aggregation network based on the improvement of VoVNet. Compared with the most commonly used target detection backbone network ResNet, it has higher feature extraction capabilities at a lower computational cost. The neck network is responsible for feature fusion and extraction. The neck network of the original YOLOv7 network constructs a feature pyramid (SPPCSPC) to extract the feature layers of the middle layer, the middle lower layer and the bottom layer of the backbone network to achieve multi-scale feature fusion. However, in the process of downsampling the feature layer, some detail information will be lost. The head network adjusts the number of channels and predictions through the re-parameterized structure according to the obtained feature maps of three sizes: large, medium and small, and then obtains the final prediction result through the CIOU (Complete intersection over union) loss function and nonlinear maximum suppression (Non-Maximum Suppression, NMS) processing.

[0084] Figure 2 The figure is the existing YOLOv7 network model diagram. The image to be detected is first uniformly scaled to 640×640×3 according to the set hyperparameters. The backbone network consists of CBS convolution, Efficient Long-range Attention Networks (ELAN) and MP modules. Among them, the MP module is composed of convolution operation added on the basis of pooling (Maxpool), which integrates the feature information after pooling with the feature information after standard convolution processing to improve the generalization of the network. The neck network consists of SPPCSPC module, MP module and ELAN' module structure. Among them, the SPPCSPC module is composed of Spatial Pyramid Pooling (SPP) and Cross Stage Partial Channel (CSPC), which is used to extract the feature layers of the middle layer, lower middle layer and bottom layer of the backbone network to realize the fusion of multi-scale features. However, in the process of downsampling the feature layer, some detail information will be lost. The head network consists of a re-parameterized convolution (Re-parameterized Convolution, RepConv) and a detection head Detect.

[0085] Figure 3 The schematic diagram of SPDConv when scale=2. The feature size of the original input image after feature extraction is (S, S, C1). Figure XFirst, a series of sub-feature maps f with sizes of (S / 2, S / 2, C1) are obtained through the SPD layer x，y , which is a downsampling operation. Then the sub-feature maps are connected along the channel dimension to obtain an intermediate feature with a size of (S / 2, S / 2, 4C1) Figure X '. Finally, the non-strided convolutional layer is used to filter the final feature with a size of (S / 2, S / 2, C2) Figure X ”.

[0086] Figure 4 This is the structure diagram of the polarized self-attention mechanism (PSA). It has two branches, one branch is used for the self-attention mechanism in the channel dimension, and the other branch is used for the self-attention mechanism in the spatial dimension. Finally, the results of these two branches are fused to obtain the output of the polarized self-attention structure. There are two modes of polarized attention mechanism, one is series type and the other is parallel type.

[0087] Figure 5 To improve the YOLOv7 network model, we added the NWD loss function to the original YOLOv7 network model, added the SPDConv module before the 20×20 image detection head, and added the polarization self-attention mechanism before downsampling. Figure 5 Four groups of experiments were conducted to add PSA attention modules to the three different positions marked in the network, ①, ②, and ③. The different added positions, quantities, modes, and corresponding network names are shown in Table 1. Position ① is before three downsamplings of the backbone network, and positions ② and ③ are before one downsampling of the neck network. The fourth group of experiments selected the series-parallel structure of the attention mechanism at the optimal position in the first three groups of experiments.

[0088] Table 1 Adding attention modules at different positions

[0089] network PSA Add Location PSA quantity mode PSA Mode YOLOv7 none 0 none YOLOV7-Attention1 ① Location 3 Parallel type YOLOv7-Attention2 ② Location 2 Parallel type YOLOv7-Attention3 ②、③ Position 3 Parallel type YOLOv7-Attention4 The best position above The optimal number of Series type

[0090] Figure 6 The images are of ten types of metal surface defects, including cracks, plaques, inclusions, indentations, punchings, oxidized surfaces, rolled-in oxide scales, scratches, stains, and silk spots, from three public datasets: NEU-DET, GC10 steel surface image dataset, and Tianchi aluminum profile defect dataset. Figure 7 The photos of the motor product production and inspection production line are used to obtain scratches, pitting, rolling marks and other defect images as a self-made metal surface defect dataset. Some images are as follows Figure 8As shown. Considering that the number of images with different defects in different data sets is different, the images in the data set are expanded by translation, mirroring and chromaticity changes to balance the number of images with different defects. At the same time, the mosaic data enhancement function of YOLOv7 is used, that is, 4 pictures are randomly selected in the data set, cropped, scaled and re-stitched to form a new picture, so that there are multiple defect types in one picture, which enhances the robustness of the network for different surface defect detection. The data set has a total of 7266 images, which are randomly divided into training set, test set and validation set according to 8:1:1.

[0091] The software and hardware used in the experimental platform of this embodiment are shown in Table 2.

[0092] Table 2 Experimental software and hardware configuration table

[0093]

[0094] This embodiment uses accuracy (Precision), recall (Recall), average precision (AP) and mean average precision (mAP) as the main evaluation indicators. Recall, accuracy, average precision, average precision of all categories and floating point operations (FLOPs) are used as model evaluation indicators, and the formulas are as follows:

[0095]

[0096] FLOPs = (2C in K 2 +1)HWC out

[0097] Among them, TP (true positives) indicates the number of samples where metal surface defects are detected and defects also exist in the actual image; FP (false positives) indicates the number of samples where defects are detected but no defects exist in the actual image; FN (false negatives) indicates the number of samples where defects are not detected but defects exist in the actual image.

[0098] J (Precision, Recall) is the average precision function, which is calculated as the area under the PR curve composed of the precision rate Precision and the recall rate Recall. The average precision of all categories mAP is obtained by taking the average of the AP values ​​of all categories, i is the number of detections, c is the number of detection categories, and AP(i) is the average precision of the i-th category defect. in is the number of input channels, K is the convolution kernel size, H and W are the sizes of the output feature maps, and Cout The present embodiment uses GFLOPs as an evaluation index to reflect the computational complexity of the model, which represents the total number of floating-point operations in units of 1 billion. 1 GFLOPs = 1×10 9 FLOPs.

[0099] The ablation experiment of the proposed method and the original YOLOv7 model was carried out on the metal surface defect dataset to confirm the optimal position, number and mode of adding modules in Table 1. The accuracy, recall rate, average accuracy and floating-point operation number of different networks in Table 1 are shown in Table 3.

[0100] Table 3 Experimental comparison of different positions and numbers of attention mechanism modules

[0101] network Precision / % Recall / % mAP / % GFLOPs YOLOv7 78.6 80.1 86.0 105.3 YOLOV7-Attention1 86.4 75.9 85.2 121.0 YOLOv7-Attention2 80.4 83.1 86.1 105.9 YOLOv7-Attention3 87.2 80.5 87.7 106.3 YOLOv7-Attention4 83.1 76.9 84.4 106.3

[0102] As can be seen from Table 3, YOLOv7-Attention2 and 3, which add polarized self-attention mechanism modules at the neck ② and ③ of the YOLOv7 network model, have improved accuracy, recall and average accuracy of all categories compared with the original YOLOv7 network. Although there is a slight increase in the amount of model calculation, it is lower than the YOLOv7-Attention1 model with the attention mechanism added to the backbone network, and higher than the average accuracy of all categories of YOLOv7-Attention4 using the series polarized self-attention mechanism. The backbone network is mainly responsible for feature extraction, and the introduction of the attention mechanism may increase the possibility of erroneous feature extraction. The series polarized attention mechanism requires the superposition of two branches, which increases the computational complexity. YOLOv7-Attention3 has improved the average accuracy of all categories compared with YOLOv7-Attention2, with an increase in accuracy but a decrease in recall, and a slight increase in model complexity.

[0103] Taking into account the accuracy and recall rate of the network model, the F1 confidence score is introduced, and its calculation formula is:

[0104]

[0105] The value of the F1 confidence score is between [0,1], and the closer it is to 1, the better the model performance. Fig. 9 The F1 curves of the YOLOv7 original network, YOLOv7-Attention2, and YOLOv7-Attention3, where (a) is the YOLOv7 original network, (b) is the YOLOv7-Attention2, and (c) is the YOLOv7-Attention3. Fig. 9From the F1 confidence score curve, we can see that the F1 value of the YOLOv7-Attention3 network is high and stable, and performs well in terms of comprehensive accuracy and recall. Fig.10 It is the Loss value curve of the original YOLOv7 network, YOLOv7-Attention2 and YOLOv7-Attention3. As can be seen from the figure, the final fluctuations of each network are small, indicating that the stability of each network is good. Among them, the Loss values ​​of Attention2 and 3 with the addition of polarization self-attention modules are lower than those of the original YOLOv7 network, and have better target detection performance. Therefore, considering the F1 confidence score, Loss value, mAP and GFLOPs, the present invention adopts the YOLOv7-Attention3 network, that is, the improved algorithm of adding three polarization self-attention mechanism modules at the positions ②③ of the YOLOv7 neck network is the final attention mechanism introduction algorithm for visual detection of metal surface defects.

[0106] In order to verify the effectiveness of the improved methods of NWD, SPDConv and PSA and their combination, ablation experiments of different networks are carried out. The environment and parameter settings are kept uniform, the batch size is set to 4, and the number of epoches is set to 300. The ablation experiment results are shown in Table 4.

[0107] Table 4 Ablation experiment

[0108] network Precision / % Recall / % mAP / % GFLOPs YOLOv7 78.6 80.1 86.0 105.3 YOLOv7+NWD 87.2 78.2 86.8 105.3 YOLOv7+SPDConv 87.6 78.9 87.6 102.3 YOLOv7+PSA 87.2 80.5 87.7 106.3 YOLOv7+NWD+SPDConv 82.5 82.9 86.3 102.3 YOLOv7+NWD+PSA 86.7 81.9 87.8 106.3 YOLOv7+SPDConv+PSA 87.4 79.8 88.0 103.3 YOLOv7+NWD+SPDConv+PSA 83.0 84.4 88.3 103.3

[0109] The mAP of the YOLOv7 algorithm before improvement is 86.0% as the comparison benchmark. As can be seen from Table 3, although adding PSA slightly increases the amount of calculation, adding NWD does not increase the amount of network calculation, and embedding SPDConv reduces the amount of calculation. All three modules can improve the average accuracy of network detection. Adding the three modules can reduce the amount of network calculation while increasing mAP by 2.3%.

[0110] In order to verify the advantages of the improved YOLOv7 network model in metal surface defect detection, the same data set and parameter settings are used to compare the results of two other groups of improved methods in recent years as shown in Table 5. Experiment 1 is a method for steel plate surface defect detection based on improved YOLO v7 (Zhang Jiaxiong, Zeng Shuiling, Li Zhaoxian. Method for steel plate surface defect detection based on improved YOLOv7 [J / OL]. Control Engineering, 1-11 [2024-08-01]. https: / / link.cnki.net / doi / 10.14107 / j.cnki.kzgc.20231034), replacing CIOU with Wise-IOU loss function, adding CBAM attention mechanism in YOLOv7 neck network, and replacing YOLOv7 coupling head with decoupling head learned by implicit knowledge. This paper compares the impact of Wise-IOU versions on the detection effect and selects the optimal version; it compares SparseR-CNN, YOLOv5 and other single-step and two-step detection algorithms on the public dataset GC10 steel surface image dataset, proving the accuracy advantage of the improved algorithm. In addition, this paper uses the PCB-Master public dataset alone to verify the versatility of the improved algorithm on completely different defect datasets. Experiment 2 is to improve the small sample steel plate surface defect detection algorithm of YOLOv7 (Dou Zhi, Hu Chenguang, Li Qinghua, et al. Improved YOLOv7 small sample steel plate surface defect detection algorithm [J]. Computer Engineering and Applications, 2023, 59(23): 283-292. https: / / link.cnki.net / urlid / 11.2127.TP.20230814.1553), add SE attention mechanism module to the YOLOv7 backbone network, replace the Wise-IOU loss function, and add ACmix attention module to the neck network. This paper uses a self-made steel plate dataset to conduct an ablation experiment on the added module, and compares it with other algorithms in the YOLO series, proving the advantages of the improved YOLOv7 algorithm in detection accuracy. At the same time, this paper uses the COCO dataset to verify the versatility of the improved algorithm in various target detection tasks.

[0111] Table 5 Comparison of different improved algorithms

[0112] network Precision / % Recall / % mAP / % GFLOPs Experiment 1 77.9 80.8 84.7 153.2 Experiment 2 82.6 78.4 85.1 104.5 This paper improves the algorithm 83.0 84.4 88.3 103.3

[0113] In Table 4, the decoupling head of Experiment 1 divides the position and category information into two branches for separate processing, which increases the amount of calculation and is prone to overfitting. Experiment 2 adds different attention modules to the backbone network and the neck network, which has good performance in terms of average accuracy and amount of calculation. The improved algorithm based on YOLOv7 proposed in this paper achieves better detection accuracy and amount of calculation on this dataset compared with the other two improved algorithms. By comparison, it can be concluded that the improved YOLOv7 proposed in this invention achieves the highest detection accuracy.

Claims

1. A method for detecting metal surface defects, characterized in that: The steps include: Establish a metal surface defect image dataset; Preprocess metal surface defect images to ensure that multiple defect types exist in the same image; Improve the YOLOv7 network model; Use the improved YOLOv7 model to train and evaluate the metal surface defect dataset, and select appropriate hyperparameters during the training process; Analyze and judge the data after model training, and adjust and optimize the model; The optimal model weights are retained and the trained model is experimentally tested.

2. The metal surface defect detection method according to claim 1, characterized in that: The establishing of the metal surface defect image data set comprises: By using the surface defect images of metal parts actually collected on the production line and combining them with public image datasets, we can realize the image collection of various types of metal surface defects in different environments; mark the location and category information of each surface defect in the image, expand the images in the dataset, and randomly divide them into training set, test set and validation set.

3. The metal surface defect detection method according to claim 2, characterized in that: The image acquisition of various types of metal surface defects under different environments includes: The public NUE-DET, GC10 steel surface image dataset, Tianchi aluminum profile defect dataset and the self-made dataset of metal parts surface defect images were used to screen the typical defects on the metal surface, and target detection was performed for 10 types of surface defects, including cracks, plaques, inclusions, indentations, punchings, oxidized surfaces, rolled-in oxide scales, scratches, stains and silk spots.

4. The metal surface defect detection method according to claim 2, characterized in that: The position and category information of each surface defect in the image is marked as follows: Use the LabelImg annotation tool to mark the location and category information of each metal surface defect in yolo format; The randomly dividing the images in the data set into a training set, a test set and a validation set after expanding the images in the data set includes: The images in the dataset are expanded by translation, mirroring and chromaticity changes, and randomly divided into training set, test set and validation set in the ratio of 8:1:

1.

5. The metal surface defect detection method according to claim 1, characterized in that: The preprocessing of the metal surface defect image comprises: The detection model loads the input metal surface image into memory, sets mosaic data enhancement, randomly crops, scales and reassembles four images to form a new image, so that there are multiple defect types in one image, and enhances the robustness parameters of the network for different surface defect detection; sets the input image size to be uniformly processed as 640×640; automatically verifies and modifies the initial anchor frame size parameters so that the best recall rate of the anchor frame on the metal surface defect dataset is greater than or equal to 0.

98.

6. The metal surface defect detection method according to claim 1, characterized in that: The improved YOLOv7 network model includes: The metal surface defect image is input into the backbone network for feature extraction, feature fusion is performed through the neck network, non-maximum suppression operation is performed through the head network, and the detection result is output; based on the CIOU loss function, the normalized Wasserstein distance loss function is introduced to improve the positioning accuracy of small-sized targets; the SPDConv module is added before the small-sized prediction head to improve the accuracy of small targets and obtain effective metal surface defect features; the polarization self-attention module is added before the downsampling operation of the neck network to enhance the network feature extraction capability.

7. The metal surface defect detection method according to claim 6, characterized in that: Based on the CIOU loss function, the normalized Wasserstein distance loss function is introduced, including: Based on the CIOU loss function used in YOLOv7, the normalized Wasserstein distance loss function is introduced as a metric, that is, the rectangular box is modeled as a Gaussian distribution, and then the distribution distance is used to measure the similarity between the rectangular boxes; the Wasserstein distance can reflect W2 when the two distributions do not overlap or the overlapping part is negligible. 2 (N a ,N b ) distribution distance; The normalized Wasserstein distance NWD is calculated as follows: C is a constant related to the data set, N a and N b They are respectively the real box A=(c xa ,c ya ,ω a ,h a ) and prediction box B = (c xb ,c yb ,ω b ,h b ) modeled Gaussian distribution, W2 2 (N a ,N b ) is the Wasserstein distance of the Gaussian distribution modeled for the annotation box; The ratio between CIOU and NWD loss is adjusted to 5:5, and the sum of the two is used as the final loss function value to balance the requirements of defect detection tasks of different sizes.

8. The metal surface defect detection method according to claim 6, characterized in that: The adding of the SPDConv module before the small-size prediction head includes: The SPDConv module consisting of a space-to-depth layer and a non-strided convolution layer is added before the small-size image prediction head to reduce image information loss; for the original input image, the feature map X is obtained after feature extraction, and a series of sub-feature maps f are first obtained through the SPD layer. x，y , which is a downsampling operation; then the sub-feature maps are connected along the channel dimension to obtain the intermediate feature map X'; finally, the non-strided convolution layer is used to filter the final feature map X".

9. The metal surface defect detection method according to claim 6, characterized in that: The adding of a polarization self-attention module before the downsampling operation of the neck network includes: Three parallel polarized self-attention mechanisms are added before the two downsampling of the YOLOv7 neck network to keep the network at a high resolution in the channel and spatial dimensions.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images