Object surface defect detection method based on yolov7

Abstract

This invention belongs to the field of computer vision, such as surface defect detection. Based on research into the characteristics of surface defects and the YOLOv7 algorithm architecture, a YOLOv7-based surface defect detection method is proposed, including: acquiring images, constructing training sets, test sets, and validation sets, and building a surface defect recognition model based on the YOLOv7 model: adding the EVCBlock module after the SPPCSPC module; replacing the Concat module with the AScat module; adding the SPDConv module to the output of the feature extraction network; and using the trained surface defect detection model to identify defects in the image to be tested. This invention enables the model to more accurately detect and locate small target defects, thereby improving the overall accuracy of surface defect detection, especially small target defect detection.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision such as surface defect detection, and particularly relates to a method for detecting surface defects of an object based on YOLOv7. Background Art

[0002] Surface defect detection is crucial to modern manufacturing, improving production efficiency and quality. Defective items are inevitable due to raw materials and environmental factors. Defects can impact product quality, lifespan, and safety. Detection can improve product quality and safety, reduce the occurrence of problems and hidden dangers, protect user rights, and reduce waste. In the past, surface defect detection relied primarily on manual labor, but manual inspection is time-consuming, labor-intensive, and inefficient, making it difficult to meet the demands of high-efficiency, large-scale production. With the advancement of computer technology and the widespread application of deep learning algorithms, automated surface defect detection methods based on image processing and machine learning are gaining increasing attention. Leveraging computer vision technology and machine learning algorithms, high-precision and high-speed detection of surface defects is possible, effectively improving product quality and safety while also increasing production efficiency, reducing the number of defective products, and lowering production costs.

[0003] Before the advent of deep learning algorithms, surface defect detection generally relied on traditional methods such as machine vision. These methods are susceptible to background interference, have difficulty detecting subtle defects, and are insensitive to variations in defect size and shape. While various object detection algorithms based on image processing and machine learning, such as SSD and Faster RCNN, have been widely applied to surface defect detection, these algorithms have demonstrated accurate positioning but low classification accuracy. The introduction of the YOLO algorithm in 2016 has further improved both object detection accuracy and speed. With the continuous development of the YOLO family of algorithms, an increasing number of models in the defect detection field are using them as their foundation. This has significantly increased detection speed, meeting the balanced accuracy and speed requirements of real-world projects. However, most models have shown limited improvement in small object detection, leaving room for improvement in surface defect detection accuracy. Summary of the Invention

[0004] The reason why small surface defects are difficult to detect is that they have uncertain positions and shapes, which makes it easy for the target detection model to lose or ignore the feature information of small targets when extracting and fusing surface defect feature information, resulting in false detection or missed detection of small targets.

[0005] To address the above technical issues, the present invention adopts the following technical solutions: Using YOLOv7 as the base network, the EVC module is first used to complement the extracted feature information, thereby enhancing the feature information of small targets. Based on the optimization points existing in the feature extraction network PAFPN, a new adaptive fusion module AScat is proposed in combination with the spatial adaptive attention mechanism to replace the concatenation module (Concat) in the PAFPN feature extraction network, enabling the model to more accurately detect and locate small target defects. Finally, SPDconv, specifically designed for predicting small targets, is introduced at the output end to prevent small target feature information from being lost during the convolution process, thereby improving the overall accuracy of the improved YOLOv7 in surface defect detection, especially small target defect detection.

[0006] A surface defect detection method based on yolov7 is carried out in the following steps:

[0007] Step (1): Get pictures and build training set, test set, and validation set

[0008] Step (2): Based on the YOLOv7 model, build a surface defect recognition model:

[0009] Add the EVCBlock module after the SPPCSPC module of the YOLOv7 model;

[0010] The Concat module of the YOLOv7 model is replaced with the AScat module; the Concat module is used for hierarchical feature fusion in the YOLOv7 model;

[0011] Adding an SPDConv module to the network output end of the YOLOv7 model to obtain a surface defect recognition model;

[0012] Step (3) training the surface defect detection model constructed in step (2);

[0013] Step (4) uses the surface defect detection model trained in step (3) to perform defect recognition on the image to be tested.

[0014] Preferably, in step (2), adding an EVCBlock module after the SPPCSPC module of the YOLOv7 model comprises the following steps:

[0015] First, write the EVCBlock module code into the common.py configuration file of the Yolov7v7 model, then declare the EVCBlock module in the yolo.py file, and then add the EVCBlock module after the SPPCSPC module on the 51st structure line of the yolov7.yaml file of the Yolov7v7 model. Name the updated yolov7.yaml file better_yolov7.yaml file;

[0016] The EVCBlock module includes a lightweight MLP and a learnable visual center mechanism LVC. The lightweight MLP is used to capture the global long-range dependencies of features, and the learnable visual center mechanism LVC is used to aggregate local area features within the layer.

[0017] Preferably, in step (2), the Concat module of the YOLOv7 model is replaced with an AScat module, which specifically includes the following steps:

[0018] First, write the AScat module code into the common.py configuration file of the Yolov7 model, then declare the AScat module in the yolo.py file, and then replace the Concat module in the 56th and 68th structure lines in the better_yolov7.yaml file with the AScat module, and add the AScat module after the Concat module in the 81st and 94th structure lines to obtain the updated better_yolov7.yaml file.

[0019] The AScat module takes the feature map as input, and the fusion formula of the AScat module is as follows:

[0020]

[0021] Among them, y ij is the weighted fusion of the feature map at (i, j) after feature fusion, and are the feature vectors at (i, j) after the feature map is magnified after being transferred from bottom to top and from left to right before feature fusion, respectively, ij and β ij is the coefficient, and α ij +β ij =1 and α ij， β ij ∈[0,1]; use and To control the parameters, and It is automatically generated by backpropagating from the input feature map using 1×1 convolution;

[0022] Preferably, in step (2), adding an SPDConv module to the network output end of the original YOLOv7 model specifically includes the following steps:

[0023] First, write the SPDConv module code into the common.py configuration file of the Yolov7 model, then declare the SPDConv module in the yolo.py file. In the updated better_yolov7.yaml file, add the SPDConv module after RepConv on line 106.

[0024] The SPDconv includes: a space-to-depth layer SPDlayer and a non-strided convolution layer Non-stridedlayer; the space-to-depth layer SPDlayer implements downsampling of the feature map while retaining feature information through frame cyclic super-resolution conversion;

[0025] Preferably, the step (3) includes setting training parameters:

[0026] The training batch size is set to Batch=12, the learning rate is set to Ir=0.01, and the number of training iterations is Epoch=300. After the training is completed, the training results are: the accuracy rate is improved to 77.11%, the recall rate is improved to 71.22%, and the mAP_0.5 is improved to 78.35%.

[0027] Preferably, in step (1), the image is derived from the steel surface defect dataset NEU-DET;

[0028] The constructing of the training set, the test set, and the validation set includes: using the prepare_train_val.py program to randomly divide the image dataset into the training set, the test set, and the validation set in a ratio of 7:2:1.

[0029] The improvements of the yolov7 model of the present invention mainly include:

[0030] (1) Use the EVC module to improve the YOLOv7 feature fusion network

[0031] YOLOv7's feature fusion network uses the aggregated feature pyramid (FAFPN) network architecture. PAFPN introduces features of different scales for fusion by establishing information flow channels with different paths, thus achieving more effective cross-scale information interaction. However, FAFPN only focuses on the interaction of inter-layer features and ignores the intra-layer feature rules, which are very useful. Therefore, the EVC module is used to correct the feature map before feature fusion. Improving the information of the feature map can effectively improve the result of feature map fusion. The structure and effect of the EVC module are shown in the figure. Figure 2shown.

[0032] (2) Replace the Concat module with the adaptive feature fusion module ASCAT to improve the performance of the YOLOv7 feature fusion network.

[0033] The AScat module's fusion method differs from the Concat module. Unlike the Concat module, which directly concatenates features, the AScat module first amplifies the input feature map channels, multiplies the amplified feature map by the generated weights, and then directly adds them along the channels. The AScat module can be viewed as an attention mechanism that focuses on the scale and semantic level of image objects. Its core concept is to use a spatial attention mechanism based on the input feature map to generate weights for integration.

[0034] (3) Using SPDConv to improve the small target defect detection performance of YOLOv7

[0035] SPDconv is a convolutional building block that excels at detecting small objects. Steel surface defects often contain tiny defects. We chose to add an SPDconv to the output of the deepest layer of YOLOv7. This is because small defects are often detected at the deepest layer. Therefore, using SPDconv at the deepest layer effectively improves the performance of detecting small defects while barely increasing the computational burden.

[0036] Compared with the prior art, the surface defect detection method of an improved YOLOv7 in the present invention can achieve the following beneficial effects:

[0037] The EVC module is used to complete the extracted feature information, allowing the feature map to perform better in subsequent feature fusion. Based on the optimization points in the feature extraction network PAFPN, a new adaptive fusion module AScat is proposed in combination with the spatial adaptive attention mechanism to replace the splicing module in the PAFPN feature extraction network. The AScat module can accurately and effectively identify and fuse small target features. Using these two improved methods, the situation where small target feature information is often overlooked in feature fusion is improved. SPDconv, which is specifically designed for predicting small targets, is used at the output end to ensure that small target feature information is effectively retained during convolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Improved yolov7 model architecture diagram

[0039] Figure 2 EVC module structure diagram

[0040] Figure 3 Improved YOLOv7 feature fusion network without AScat module

[0041] Figure 4 Improved YOLOv7 feature fusion network using AScat module

[0042] Figure 5 Schematic diagram of AScat module feature fusion

[0043] Figure 6 Improved yolov7 model training results

[0044] Figure 7 Improved yolov7 model validation set test results Specific implementation methods

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following describes the present invention in combination with specific implementations and drawings.

[0046] The surface defect detection model method of the improved YOLOv7 provided by the present invention includes:

[0047] Step (1): Use the prepare_train_val.py program to randomly divide the steel surface defect dataset NEU-DET provided by Northeastern University into training set, test set, and validation set in a ratio of 7:2:1.

[0048] Step (2): Build an improved YOLOv7 surface defect recognition model. The specific steps are as follows:

[0049] (1) Add the EVC module after the SPPCSPC module of the original YOLOv7 model.

[0050] First, add the EVCBlock module code to the common.py configuration file of the Yolov7 model, then declare the EVCBlock module in the yolo.py file. Then, create the better_yolov7.yaml file and add the EVCBlock module after the SPPCSPC module on line 51 of the original Yolov7 file.

[0051] YOLOv7's feature fusion network uses the aggregated feature pyramid (FAFPN) network architecture. PAFPN introduces features of different scales for fusion by establishing information flow channels with different paths, thus achieving more effective cross-scale information interaction. However, FAFPN only focuses on the interaction of inter-layer features and ignores the intra-layer feature rules, which are very useful. Therefore, the EVC module is used to correct the feature map before feature fusion. Improving the information of the feature map can effectively improve the result of feature map fusion. The structure and effect of the EVC module are shown in the figure. Figure 2 shown.

[0052] The core of the EVC module consists of two modules: a lightweight MLP and a learnable visual center mechanism (LVC). The former is responsible for capturing global long-range dependencies of features, while the latter is responsible for aggregating local features within a layer. The lightweight MLP is used to obtain global information rather than a transformer because the two are similar in their ability to extract global information from deep features, but the lightweight MLP has lower computational complexity. LVC is an encoder with an inherent dictionary that acquires local feature information through self-learning. The image in the EVC module first passes through a Stem Block layer consisting of a 7×7 convolution with a channel size of 256, a batch normalization layer, and an activation function layer for feature smoothing. After obtaining global and local information from the feature map, this information is concatenated to form a complete feature map, which is then output. After the EVC module completes the feature map, the defect features are greatly enhanced, allowing these defects to be effectively identified by the model in subsequent processing.

[0053] (2) Replace the Concat module used for hierarchical feature fusion in the feature fusion network with the AScat module.

[0054] First, add the AScat module code to the common.py configuration file of the Yolov7 model. Then, declare the AScat module in the yolo.py file. In the better_yolov7.yaml file, replace the Concat module on lines 56 and 68 with the AScat module. Add the AScat module after the Concat module on lines 81 and 94. The line numbers mentioned above are after adding the EVC module.

[0055] Although the FAFPN network architecture fuses feature information at different levels for multi-scale learning, which alleviates the problem of cross-scale feature fusion to a certain extent, there are some problems in feature fusion that make the fusion network pay insufficient attention to the feature information of small targets, resulting in feature information loss.

[0056] like Figure 3 As shown in the figure, PAFPN uses a direct concatenation method during feature fusion, which is equivalent to a fixed-weight fusion method. However, the optimal fusion ratio depends on the size of the objects in the input image. Obviously, the size of objects varies from image to image, and the size distribution of objects in an image depends on the dataset. The fixed weights used to integrate the bottom-up and top-down feature maps in PAFPN are equivalent to a uniformly weighted average of the two. This may result in insufficient attention to the characteristics of small defects during the defect feature fusion process, resulting in missing feature information for small defects.

[0057] The improved YOLOv7 feature fusion network without AScat module improves the target positioning and classification accuracy, and proposes a new adaptive feature fusion module AScat to replace the ordinary feature channel fusion module Concat (such as Figure 4 ), this module automatically calculates spatially adaptive weights using the input feature information from left to right and the vertical feature information (in the example of the image, this is the feature information transmitted from bottom to top) and then performs adaptive feature fusion based on the weights. For example, if an image of a small object is input, the trained scale attention module in the path will adaptively weight the feature map based on the object scale contained in the input image, thereby emphasizing higher-resolution feature maps. After feature fusion, the characteristics of small object defects receive more attention.

[0058] The AScat module's fusion method differs from the Concat module, which directly concatenates features. The AScat module first amplifies the input feature map channels, multiplies the amplified feature map by the generated weights, and then directly adds them along the channels. The AScat module can be viewed as an attention mechanism that focuses on the scale and semantic level of image objects. Its core concept is to use a spatial attention mechanism based on the input feature map to generate weights for integration. Through the weighted fusion of the vectors at the feature map (i, j), the coefficients are adaptively learned by the network and are shared across all channels. Figure 5 The fusion diagram is shown below. The fusion formula of ASCAT is as follows:

[0059]

[0060]

[0061] y in the formula ij It is the weighted fusion at the feature map (i, j) after feature fusion, and They are the feature vectors at (i, j) after the feature maps are magnified after being transferred from bottom to top and from left to right before feature fusion, α ij and β ij is the coefficient, and α ij +β ij =1 and α ij， β ij ∈[0,1] (can be achieved through the softmax function). Here we use and To control the parameters, and It is automatically generated by backpropagating from the input feature map using 1×1 convolution. This method can effectively solve the inconsistency problem of fused images and highlight the desired target points.

[0062] (3) Add the SPDConv module to the output of the feature extraction network.

[0063] First, add the SPDConv module code to the common.py configuration file of the Yolov7 model. Then, declare the SPDConv module in the yolo.py file. In the better_yolov7.yaml file, add the SPDConv module after RepConv on line 106. The number of structure lines mentioned above is after adding the EVC and AScat modules.

[0064] SPDconv is a convolutional building block that excels at detecting small objects. Surface defects often contain tiny defects, so we chose to add SPDconv to the output of the deepest network layer. This is because small defects are often detected at the deepest network output. Therefore, using SPDconv at the deepest network output effectively improves the performance of detecting small defects while barely increasing the computational burden.

[0065] SPDconv consists of a spatial-to-depth layer (SPDlayer) and a non-strided convolutional layer (Non-stridedlayer). SPDlayer uses frame-based recurrent super-resolution technology to downsample feature maps while preserving as much feature information as possible. This allows for more accurate identification of small target defects in subsequent detection and recognition, effectively improving the model's detection performance for small target defects.

[0066] Step (3) trains the improved YOLOv7 surface defect detection model constructed in step (2). The specific steps are as follows:

[0067] Build a training framework on the server, set the training parameters, set the training batch size to Batch = 12, the learning rate to Ir = 0.01, and the number of training iterations to Epoch = 300. After the training is completed, the training results show that the accuracy rate is improved to 77.11%, the recall rate is improved to 71.22%, and the mAP_0.5 is improved to 78.35%.

[0068] Step (4) uses the improved YOLOv7 surface defect detection model trained in step 3 to perform target recognition on the verification set.

[0069] In the training framework, the test parameter batch size was set to Batch = 36, and the test weights used the best model weights. The test results on the validation set showed an improvement in accuracy to 91.4%, recall to 86.5%, and mAP_0.5 to 93.7%.

Claims

1. The object surface defect detection method based on yolov7 is characterized by: The following steps are involved: Step (1): Obtain images and construct training set, test set, and validation set; Step (2): Based on the YOLOv7 model, build a surface defect recognition model: An EVCBlock module is added after the SPPCSPC module of the YOLOv7 model; the EVCBlock module includes a lightweight MLP and a learnable visual center mechanism LVC; The Concat module of the YOLOv7 model is replaced with the AScat module; the Concat module is used for hierarchical feature fusion in the YOLOv7 model; the AScat module takes the feature map as input, and the fusion formula of the AScat module is as follows: Among them, y ij is the weighted fusion of the feature map at (i, j) after feature fusion, and are the feature vectors at (i, j) after the feature map is magnified after being transferred from bottom to top and from left to right before feature fusion, respectively, ij and β ij is the coefficient, and α ij +β ij =1 and α ij , β ij ∈[0,1]; use and To control the parameters, and It is automatically generated by backpropagating from the input feature map using 1×1 convolution; Adding an SPDConv module to the output end of the feature extraction network of the YOLOv7 model to obtain a surface defect recognition model; The SPDconv includes: a space-to-depth layer SPDlayer and a non-strided convolution layer Non-stridedlayer; the space-to-depth layer SPDlayer implements downsampling of the feature map while retaining feature information through frame cyclic super-resolution conversion; Step (3) training the surface defect detection model constructed in step (2); Step (4) uses the surface defect detection model trained in step (3) to perform defect recognition on the image to be tested.

2. The object surface defect detection method based on yolov7 according to claim 1, characterized in that: In the step (2), the EVCBlock module is added after the SPPCSPC module of the YOLOv7 model, including the following steps: First, write the EVCBlock module code into the common.py configuration file of the Yolov7v7 model, then declare the EVCBlock module in the yolo.py file, and then add the EVCBlock module after the SPPCSPC module on the 51st structure line of the yolov7.yaml file of the Yolov7v7 model. Name the updated yolov7.yaml file better_yolov7.yaml file.

3. The object surface defect detection method based on yolov7 according to claim 2, characterized in that: In step (2), the Concat module of the YOLOv7 model is replaced with the AScat module, which specifically includes the following steps: First, write the AScat module code into the common.py configuration file of the Yolov7 model, then declare the AScat module in the yolo.py file, and then replace the Concat module in the 56th and 68th structure lines with the AScat module in the better_yolov7.yaml file. Add the AScat module after the Concat module in the 81st and 94th structure lines to obtain the updated better_yolov7.yaml file.

4. The object surface defect detection method based on yolov7 according to claim 3, characterized in that: In the step (2), an SPDConv module is added to the feature extraction network output portion of the YOLOv7 model, specifically comprising the following steps: First, write the SPDConv module code into the common.py configuration file of the Yolov7 model, and then declare the SPDConv module in the yolo.py file; in the updated better_yolov7.yaml file, add the SPDConv module after RepConv on line 106 of the structure.

5. The object surface defect detection method based on yolov7 according to claim 4, characterized in that: The step (3) includes setting the training parameters: The training batch size is set to Batch=12, the learning rate is set to Ir=0.01, and the number of training iterations Epoch=300.

6. The object surface defect detection method based on yolov7 according to claim 5, characterized in that: In the step (1), the image is derived from the steel surface defect dataset NEU-DET; The constructing of the training set, the test set, and the validation set includes: using the prepare_train_val.py program to randomly divide the image dataset into the training set, the test set, and the validation set in a ratio of 7:2:1.

Images