Car door surface defect detection method based on improved YOLOv8 network

Abstract

The present invention discloses a vehicle door surface defect detection method based on an improved YOLOv8 network, comprising: collecting vehicle door surface images, customizing a data set, performing image preprocessing on the data set, performing data enhancement on pinhole defect images and sheet metal defect images, and proportionally dividing the method into a training set, a validation set, and a test set; training the improved YOLOv8 network with the training set, and verifying it with the validation set; and testing it with the test set. The present invention extracts feature information of the vehicle door surface image by improving the YOLOv8 network, and effectively overcomes the influence of the imbalance of vehicle door surface defects on network performance during the training process due to the introduction of an improved residual module, an improved feature fusion method, and an improved CBAM attention module, thereby significantly reducing the number of network parameters and the amount of calculation, achieving a balance among real-time detection, accurate detection, and lightweight, and being extendable to more defect detection scenarios. The method has broad application prospects in the industry.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and target detection, and in particular to a method for detecting surface defects of vehicle doors based on an improved YOLOv8 network. Background Art

[0002] With the rapid development of the automotive industry, the importance of a vehicle's appearance to consumers is growing. To meet these demands, automakers are placing increasing emphasis on inspecting the exterior quality of their vehicles. Door surface defect detection, a crucial component of the vehicle, plays an indispensable role in exterior inspection. However, traditional door surface defect inspection methods rely on manual inspections, which are not only inefficient but also significantly reduce accuracy and speed after extensive manual inspections.

[0003] With the rapid development of deep learning, especially advancements in object detection networks (ODNs), methods based on these networks have attracted widespread attention in the field of door surface defect detection. Many automotive production lines have begun to employ robotic arms and cameras to automatically detect door surface defects. However, the lack of high-performance computing equipment on production sites has limited the practical application of ODNs. Furthermore, most ODNs currently used on automotive production lines sacrifice accuracy for real-time detection, effectively serving only as auxiliary detection methods. Therefore, finding a method that balances real-time detection, high accuracy, and lightweight design is urgent and has become a key research focus in the field of door surface defect detection. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem that the current target detection network is difficult to achieve a balance among real-time detection, accurate detection and lightweight in door surface defect detection. A door surface defect detection method based on an improved YOLOv8 network is provided. The improved residual module effectively reduces the number of backbone network parameters and the amount of calculation, thereby reducing its requirements on the computing power of hardware equipment, making it more adaptable to the requirements of the actual automobile production environment; at the same time, the improved feature fusion method and the improved CBAM attention module achieve a balance between real-time detection and accurate detection, thereby realizing automatic door surface defect detection.

[0005] To achieve the above purpose, the technical solution provided by the present invention is: a vehicle door surface defect detection method based on an improved YOLOv8 network. The improved YOLOv8 network is a two-part improvement on the original YOLOv8 network. The first part is to improve the residual module of the backbone network: the original two series-connected 3×3 convolutional layers are replaced by one 3×3 depth-separable convolutional layer and one 3×3 Ghost-Dilated convolutional layer to construct parallel feature extraction. Based on the idea of ​​split-transform-fusion, different features are extracted through parallel convolution operations for splicing operations, thereby reducing the number of parameters and computational complexity of the backbone network; the second part is to improve the different features of the neck network. The fusion method is improved. Instead of fusing features of two adjacent different scales, features of all scales are scaled into a feature map of a unified scale. The mean of the feature maps of different scales is calculated to obtain a mean feature map. The mean feature map is fused with features of different scales. The fused feature map is then aggregated using an improved CBAM attention module. The improvement to the CBAM attention module is to replace the original spatial attention module with a context aggregation module. The context aggregation module is a self-attention module that is used to enhance the global information of the fused features, thereby reducing the impact of imbalance on network accuracy during the training process of door surface defect detection.

[0006] The specific implementation of the vehicle door surface defect detection method includes the following steps:

[0007] 1) Collect door surface defect images, build a custom door surface defect dataset, and divide it into training set, validation set, and test set;

[0008] 2) The training set is fed into the improved YOLOv8 network in batches for training. After the images in the training set are feature extracted by the backbone network, the feature layers of different scales are fused and the extracted features are enhanced through the neck network. The network loss is obtained through the detection head, and the network parameters and weights are updated based on the gradient descent strategy and loss calculation backpropagation. In addition, the network performance is verified on the validation set after each preset number of network training. When the evaluation index on the validation set reaches the optimal value and the network loss value no longer decreases, the network with the best performance after training and verification is obtained.

[0009] 3) The test set is fed into the best performing network obtained in step 2) to obtain the classification value predicted by the network and the coordinates of the predicted bounding box, where different classification values ​​correspond to different defects.

[0010] Furthermore, in step 1), before photographing the door surface, the robot arm is first controlled to the designated photographing area, and the shape of the stripes in the stripe light source is adjusted according to different simulated production situations. After the image displayed by the host computer is stable, the host computer controls the camera to photograph the door surface defect image, and then adjusts the position of the robot arm to traverse the entire door to complete the acquisition of the door surface defect image; after the acquisition is completed, the image is preprocessed, and the preprocessing operation flow includes: first, bilinear interpolation is used to scale the original image, unify the size to a resolution of 640×640, adjust the saturation, hue and exposure of the image, and then perform Gaussian filtering on the transformed image to reduce the influence of noise in the image on the detection accuracy; using the preprocessed image For example, a car door surface defect dataset is constructed. The dataset includes normal car door surface images, pinhole defect images, scratch defect images, and sheet metal defect images. The Mosaic method is used to perform data enhancement on pinhole defect images and sheet metal defect images. The Mosaic method first randomly selects a position as the position of the center image. The position is randomly selected within an area of ​​the input image, and the selected center image is placed at the center of the input image. The boundary image is scaled, translated, and flipped and placed on the four boundaries of the input image. Finally, the target bounding box in the boundary image is adjusted according to the position and scaling ratio of the boundary image in the input image. The constructed car door surface defect dataset is divided into training set, validation set, and test set according to the set ratio.

[0011] Furthermore, in step 2), the improved YOLOv8 network includes three parts: the backbone network, the neck network, and the detection head. The details are as follows:

[0012] The first layer of the backbone network is a 3×3 convolution operation, which is used to adjust the number of channels of the input image and downsample the input size;

[0013] The second and fifth layers are both constructed with a 3×3 convolution operation followed by an improved residual module operation repeated three times. The third and fourth layers are both constructed with a 3×3 convolution operation followed by an improved residual module operation repeated six times. The improved residual module extracts features in five steps, as follows:

[0014] The first step is a 3×3 convolution operation that adjusts the channel dimension of the original input feature map to twice its original size, obtaining a new input feature map I.

[0015] The second step is to divide the input feature map I into two sub-feature maps I1 and I2 according to the channel dimension;

[0016] The third step is to perform a 3×3 depth-wise separable convolution operation on the sub-feature map I1 to form a new feature map I'1. The process is as follows:

[0017] I'1=I1*DW3 (1)

[0018] Where I'1 represents the feature map output by the 3×3 depthwise separable convolution operation, DW3 represents the 3×3 depthwise separable convolution operation. The depthwise separable convolution operation splits the standard convolution operation into two steps: depthwise convolution operation and pointwise convolution operation. The depthwise convolution operation independently performs convolution operation on each channel of the input feature map, and the pointwise convolution operation uses a 1×1 convolution kernel to perform convolution operation on the output of the depthwise convolution operation. The process is as follows:

[0019] X'1=X1*K3*K1 (2)

[0020] In the formula, X'1 represents the feature of one channel of I'1, X1 represents the feature of one channel of I1, K3 represents a 3×3 convolution operation, and K1 represents a 1×1 convolution operation;

[0021] The fourth step is to perform a 3×3 Ghost-Dilated convolution operation on the sub-feature map I2 to form a new feature map I'2. The Ghost-Dilated convolution operation splits the convolution operation into three steps: main branch convolution operation, pseudo branch convolution operation and feature fusion operation. Among them, the main branch convolution operation operates on 75% of the channels of the input feature map and uses a 3×3 convolution operation to process the input features. The pseudo branch convolution operation operates on 25% of the channels of the input feature map and uses a 7×7 hole convolution operation to process the input features. Finally, the output results of the two branches are subjected to a feature splicing operation. The process is as follows:

[0022]

[0023] Where I'2 represents the feature map output by the 3×3 Ghost-Dilated convolution operation, The characteristic of the 75% channel representing I2, Represents the characteristics of the 25% channel of I2, represents a 7×7 dilated convolution operation, and C represents a feature concatenation operation in the channel dimension;

[0024] The fifth step is to obtain a new feature map I3 by performing feature concatenation operation on the feature map I'1 and the feature map I'2, and to add the feature map I3 to the feature map I to form a new feature map I4, thereby fusing the information between the feature maps obtained by different types of convolution methods and enriching the features between different semantic information of the image. The process is as follows:

[0025] I3=C(I'1,I'2) (4)

[0026] I4=I+I3 (5)

[0027] Where, I3 represents the feature map obtained by concatenating the feature map I'1 and the feature map I'2, and I4 represents the feature map obtained by adding the feature map I and the feature map I3;

[0028] The neck network uses an improved feature fusion method to fuse the features F3, F4, and F5 output by the three to five layers of the backbone network. This improved feature fusion method is divided into four steps to fuse features. The details are as follows:

[0029] The first step is to scale F3 and F5 to the same scale as F4. The process is as follows:

[0030] F'3=D(F3) (6)

[0031] F'5=U(F5) (7)

[0032] Where F3' represents the scaled output of F3, F5' represents the scaled output of F5, D represents the maximum pooling operation, and U represents the bilinear difference operation;

[0033] The second step is to perform weighted averaging using F'3, F4, F'5 and the preset hyperparameters α, β, and γ. The process is as follows:

[0034] F avg =αF'3+βF4+γF'5 (8)

[0035] Where, F avg is the average feature output by the weighted average operation;

[0036] The third step is to avg The feature fusion process with F3, F4, and F5 is as follows:

[0037] P3=C(F3,F avg *K1) (9)

[0038] P4=C(F4,F avg *K1) (10)

[0039] P5=C(F5,F avg *K1) (11)

[0040] Where, P3 is F3 and F avg The output feature after fusion, P4 is F4 and F avg The output features after fusion, P5 is F5 and F avg The output features after fusion;

[0041] The fused features P3, P4, and P5 are enhanced through the improved CBAM attention module. The improved CBAM attention module not only effectively reduces the number of parameters and calculations, but also uses the pixel-level self-attention mechanism to fuse adjacent features, improves the feature extraction capability of the door surface defect image, and enhances the context information related to the door surface defect detection task, thereby achieving the goal of high-precision detection and real-time detection; Among them, the improved CBAM attention module is divided into a channel attention module and a context aggregation module. The channel attention module calculates the weight vector λ i Applied to the input features, the important channels are highlighted and the unimportant channels are suppressed by element-by-element multiplication. The specific situation is as shown in formulas (12) and (13):

[0042] λ i =σ(f c (f gap (P i ))) i=3,4,5 (12)

[0043] P' i =λ i P i i=3,4,5 (13)

[0044] Where P' i represents the output feature after the channel attention module is enhanced, σ represents the Sigmoid function, f c represents the fully connected layer, and f gap Global average pooling operation, i represents the i-th layer architecture, P i represents the output features of the i-th layer architecture, λ i Indicates P i The corresponding weight vector;

[0045] Then, P' i As input features, it is input into the context aggregation module, which is fed into the query feature map P' iq , key feature graph P' ik , value characteristic graph P' iv Attention calculation is performed on each position in the entire feature map, so that each position can interact and transfer information with other positions globally without being restricted by distance. This global interaction can help the model capture the correlation between different positions, thereby providing richer context information. The specific situation is shown in formulas (14), (15), (16), and (17):

[0046] P' iq =P' i *K1 i=3,4,5 (14)

[0047] P' iv =σ(P' i *K1) i=3,4,5 (15)

[0048]

[0049] P i ”=f matmul (P' iv ,P' iq ·P' ik ) i=3,4,5 (17)

[0050] Where, P i " represents the features enhanced by the context aggregation module, represents the Softmax function, f matmul is matrix multiplication;

[0051] The feature map enhanced by the improved CBAM attention module is input into the detection head to calculate the loss. In the detection head, the classification loss is calculated using the Binary cross-entropy loss function and the manually labeled true category value. The bounding box loss is calculated using the Distribution Focal loss function and the Ciou loss function and the manually labeled true normalized bounding box coordinate value. The classification loss and bounding box loss are weighted averaged to obtain the network loss. The specific situation is shown in formulas (18), (19), and (20):

[0052] L cls =f bce (p pred ,p gt ) (18)

[0053] L bbox =f ciou (b pred ,b gt )+ω dfl f dfl (b pred ,b gt ) (19)

[0054] L=ω cls L cls +ω bbox L bbox (20)

[0055] Where, L cls represents the classification loss, L bbox represents the bounding box loss, L represents the total loss, and p pred Indicates the probability that the network predicts the target to be a specified category, p gtIndicates the true probability that the target is a specified category, b pred represents the bounding box predicted by the network, b gt represents the true bounding box, ω dfl Represents the weight corresponding to the Distribution focal loss, ω cls Represents the weight corresponding to the classification loss, ω bbox represents the weight corresponding to the bounding box loss, f bce Represents the Binary cross-entropy loss function, f ciou represents the Ciou loss function, f dfl Denotes the Distribution focal loss function, which is calculated as shown in formulas (21), (22), and (23):

[0056] f bce (p pred ,p gt )=-(p gt log(p pred )+(1-p gt )log(1-p pred )) (twenty one)

[0057]

[0058]

[0059] Where, IOU represents b pred and b gt The intersection-over-union ratio, ρ represents b pred and b gt The Euclidean distance of b, c represents pred and b gt The diagonal length of the minimum bounding rectangle, γ gt Indicates b gt The aspect ratio, γ pred Indicates b pred The aspect ratio of the bounding box is τ, which means rounding up the coordinates of the four points of the bounding box, and ψ which means rounding down the coordinates of the four points of the bounding box;

[0060] After calculating the network loss, the gradient descent strategy and loss calculation back propagation are used to optimize and update the network parameter weights. The total number of iterations of the network training process is 500. The validation set is used to verify the performance of the network every 10 iterations. The termination condition of the training is that the performance indicators of the network in the validation set no longer change significantly and the total loss of the network no longer decreases.

[0061] Further, in step 3), the test set is sent to the network with the best performance to obtain the classification value predicted by the network and the coordinates of the predicted bounding box. When the recognized classification value is 0, the recognition result is the background, and it is determined to be a normal car door surface. When the recognized classification value is 1, the recognition result is a scratch, and it is determined that there is a scratch defect in the car door surface image. When the recognized classification value is 2, the recognition result is a pinhole, and it is determined that there is a pinhole defect in the car door surface image. When the recognized classification value is 3, the recognition result is sheet metal, and it is determined that there is a sheet metal defect in the car door surface image. Among them, the precision and recall rates of various defects during the test process are recorded, and the evaluation index of the best performance of the network is obtained through the highest precision and recall rate.

[0062] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0063] 1. The present invention achieves a balance among real-time detection, precise detection, and lightweight, and can realize accurate real-time door surface defect detection in actual industrial scenarios with limited computing power. Compared with traditional manual detection, it has faster detection speed and higher detection accuracy, and does not require manual review of the door during the detection process. It can realize automatic door surface defect detection, saving a lot of manpower and material resources.

[0064] 2. Based on the improved YOLOv8 network, the present invention applies an improved residual module in the backbone network to extract different features, so that the network converges to the value obtained by different convolution methods; introduces an improved fusion method of different features, and uses the improved CABM attention module to enhance the features, solving the problem of imbalance in the training process of vehicle door surface defect detection, which reduces the network detection accuracy.

[0065] 3. Compared with other target detection network methods, the present invention has less computational complexity and parameter quantity, can be applied in scenarios with limited computing power of hardware equipment, and achieves a balance between detection speed and detection accuracy, with better robustness. It is of great significance for the detection of surface defects of vehicle doors. Moreover, the proposed improved method can be migrated to other methods, and therefore has a wide range of extension space in other defect detection scenarios and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Flowchart of the present invention.

[0067] Figure 2This is a structural diagram of the improved YOLOv8 network; in the figure, conv represents the convolution operation, BilinearInterpolation represents the bilinear interpolation operation, STM Block represents the improved residual module, ECBAM Block represents the improved CBAM attention module, head represents the detection head, concatenate represents the feature concatenation operation in the channel dimension, and avg represents the average of features at different scales.

[0068] Figure 3 This is a structural diagram of the improved residual module; in the figure, Depthwise separate conv represents the depth-separable convolution operation, and Ghost-Dilated conv represents the ghost hole convolution operation. DETAILED DESCRIPTION

[0069] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0070] like Figure 1 As shown, this embodiment discloses a method for detecting surface defects of vehicle doors based on an improved YOLOv8 network, which includes the following steps:

[0071] 1) Before photographing the door surface, the robot arm is first controlled to the designated shooting area, and the shape of the stripes in the stripe light source is adjusted according to different simulated production situations. After the image displayed by the host computer is stable, the host computer controls the camera to photograph the door surface defect image, and then adjusts the position of the robot arm to traverse the entire door to complete the acquisition of the door surface defect image; after the acquisition is completed, the image is preprocessed. The preprocessing operation flow includes: first, bilinear interpolation is used to scale the original image, unify the size to a resolution of, adjust the saturation, hue and exposure of the image, and then perform Gaussian filtering on the transformed image to reduce the influence of noise in the image on the detection accuracy; the preprocessed image is used to construct the door surface defect image. The dataset includes normal car door surface images, pinhole defect images, scratch defect images and sheet metal defect images. The Mosaic method is used to perform data enhancement on pinhole defect images and sheet metal defect images. The Mosaic method first randomly selects a position as the position of the center image. The position is randomly selected from an area in the input image, and the selected center image is placed at the center of the input image. The boundary image is placed on the four boundaries of the input image after scaling, translation and flipping. Finally, the target bounding box in the boundary image is adjusted according to the position and scaling ratio of the boundary image in the input image; the constructed car door surface defect dataset is divided into training set, validation set and test set according to the set ratio.

[0072] 2) The training set is fed into the improved YOLOv8 network in batches for training. After the images in the training set are feature extracted by the backbone network, the feature layers of different scales are fused and the extracted features are enhanced through the neck network. The network loss is obtained through the detection head, and the network parameters and weights are updated based on the gradient descent strategy and loss calculation back propagation. It is also stipulated that the performance of the network is verified on the validation set after each specific number of network trainings. When the evaluation index on the validation set reaches the best and the network loss value no longer decreases, the network with the best performance after training and verification is obtained.

[0073] like Figure 2 As shown in the figure, the improved YOLOv8 network mainly makes two improvements. The first part is to improve the residual module of the backbone network: the original two serial convolutional layers are replaced by one depth-wise separable convolutional layer and one Ghost-Dilated convolutional layer to construct parallel feature extraction. Based on the idea of ​​"split-transform-fusion", different features are extracted through parallel convolution operations for splicing operations, thereby reducing the number of parameters and computational complexity of the backbone network; the second part is to improve the fusion method of different features of the neck network. The original fusion of two adjacent different-scale features is changed to scaling the features of all scales into a feature map of a unified scale, and then the mean of the feature maps of different scales is calculated to obtain the mean feature map. The obtained mean feature map is fused with the features of different scales, and then the improved CBAM attention module is used to aggregate the information of the fused feature map. The improvement of the CBAM attention module is to replace the original spatial attention module with a context aggregation module. The context aggregation module is a self-attention module that effectively enhances the global information of the fused features, thereby reducing the impact of imbalance on network accuracy during the training process of door surface defect detection.

[0074] The improved YOLOv8 network consists of three parts: backbone network, neck network and detection head. The details are as follows:

[0075] The first layer of the backbone network is a 3×3 convolution operation, which is used to adjust the number of channels of the input image and downsample the input size;

[0076] The second and fifth layer architectures are both a 3×3 convolution operation plus an improved residual module operation repeated 3 times, and the third and fourth layer architectures are both a 3×3 convolution operation plus an improved residual module operation repeated 6 times; among them, Figure 3 As shown in Figure 2, the improved residual module is divided into five steps to extract features, as follows:

[0077] The first step is a 3×3 convolution operation that adjusts the channel dimension of the original input feature map to twice its original size, obtaining a new input feature map I.

[0078] The second step is to divide the input feature map I into two sub-feature maps I1 and I2 according to the channel dimension;

[0079] The third step is to perform a 3×3 depth-wise separable convolution operation on the sub-feature map I1 to form a new feature map I'1. The process is as follows:

[0080] I'1=I1*DW3 (1)

[0081] Where I'1 represents the feature map output by the 3×3 depthwise separable convolution operation, DW3 represents the 3×3 depthwise separable convolution operation. The depthwise separable convolution operation splits the standard convolution operation into two steps: depthwise convolution operation and pointwise convolution operation. The depthwise convolution operation independently performs convolution operation on each channel of the input feature map, and the pointwise convolution operation uses a 1×1 convolution kernel to perform convolution operation on the output of the depthwise convolution operation. The process is as follows:

[0082] X'1=X1*K3*K1 (2)

[0083] In the formula, X'1 represents the feature of one channel of I'1, X1 represents the feature of one channel of I1, K3 represents a 3×3 convolution operation, and K1 represents a 1×1 convolution operation;

[0084] The fourth step is to perform a 3×3 Ghost-Dilated convolution operation on the sub-feature map I2 to form a new feature map I'2. The Ghost-Dilated convolution operation splits the convolution operation into three steps: main branch convolution operation, pseudo branch convolution operation and feature fusion operation. Among them, the main branch convolution operation operates on 75% of the channels of the input feature map and uses the conventional 3×3 convolution operation to process the input features. The pseudo branch convolution operation operates on 25% of the channels of the input feature map and uses the 7×7 hole convolution operation to process the input features. Finally, the output results of the two branches are subjected to feature splicing operation. The process is as follows:

[0085]

[0086] Where I'2 represents the feature map output by the 3×3 Ghost-Dilated convolution operation, The characteristic of the 75% channel representing I2, Represents the characteristics of the 25% channel of I2, represents a 7×7 dilated convolution operation, and C represents a feature concatenation operation in the channel dimension;

[0087] The fifth step is to obtain a new feature map I3 by performing feature concatenation operation on the feature map I'1 and the feature map I'2, and to add the feature map I3 to the feature map I to form a new feature map I4, thereby fusing the information between the feature maps obtained by different types of convolution methods and enriching the features between different semantic information of the image. The process is as follows:

[0088] I3=C(I'1,I'2) (4)

[0089] I4=I+I3 (5)

[0090] Where, I3 represents the feature map obtained by concatenating the feature map I'1 and the feature map I'2, and I4 represents the feature map obtained by adding the feature map I and the feature map I3;

[0091] The neck network uses an improved feature fusion method to fuse the features F3, F4, and F5 output by the three to five layers of the backbone network. This improved feature fusion method is divided into four steps to fuse features. The details are as follows:

[0092] The first step is to scale F3 and F5 to the same scale as F4. The process is as follows:

[0093] F'3=D(F3) (6)

[0094] F'5=U(F5) (7)

[0095] Where F'3 represents the scaled output of F3, F'5 represents the scaled output of F5, D represents the maximum pooling operation, and U represents the bilinear difference operation;

[0096] The second step is to perform weighted averaging using F'3, F4, F'5 and the preset hyperparameters α, β, and γ. The process is as follows:

[0097] F avg =αF'3+βF4+γF'5 (8)

[0098] Where, F avg is the average feature output by the weighted average operation. In this method, α = 0.3, β = 0.4, and γ = 0.3;

[0099] The third step is to avg The feature fusion process with F3, F4, and F5 is as follows:

[0100] P3=C(F3,F avg *K1) (9)

[0101] P4=C(F4,F avg*K1) (10)

[0102] P5=C(F5,F avg *K1) (11)

[0103] Where, P3 is F3 and F avg The output feature after fusion, P4 is F4 and F avg The output features after fusion, P5 is F5 and F avg The output features after fusion;

[0104] The fused features P3, P4, and P5 are enhanced through the improved CBAM attention module. The improved CBAM attention module not only effectively reduces the number of parameters and calculations, but also uses the pixel-level self-attention mechanism to fuse adjacent features, improves the feature extraction capability of the door surface defect image, and enhances the context information related to the door surface defect detection task, thereby achieving the goal of high-precision detection and real-time detection. Among them, the improved CBAM attention module is divided into a channel attention module and a context aggregation module. Among them, the channel attention module calculates the weight vector λ i Applied to the input features, the important channels are highlighted and the unimportant channels are suppressed by element-by-element multiplication. The specific situation is as shown in formulas (12) and (13):

[0105] λ i =σ(f c (f gap (P i ))) i=3,4,5 (12)

[0106] P' i =λ i P i i=3,4,5 (13)

[0107] Where P' i represents the output feature after the channel attention module is enhanced, σ represents the Sigmoid function, f c represents the fully connected layer, and f gap Global average pooling operation, i represents the i-th layer architecture, P i represents the output features of the i-th layer architecture, λ i Indicates P i The corresponding weight vector;

[0108] Then, P' i As input features, it is input into the context aggregation module, which is fed into the query feature map P' iq , key feature graph P' ik , value characteristic graph P' ivAttention calculation is performed on each position in the entire feature map, so that each position can interact and transfer information with other positions globally without being restricted by distance. This global interaction can help the model capture the correlation between different positions, thereby providing richer contextual information, as shown in formulas (14), (15), (16), and (17):

[0109] P' iq =P' i *K1 i=3,4,5 (14)

[0110] P' iv =σ(P' i *K1) i=3,4,5 (15)

[0111]

[0112] P i ”=f matmul (P' iv ,P' iq ·P' ik ) i=3,4,5 (17)

[0113] Where, P i " represents the features enhanced by the context aggregation module, represents the Softmax function, f matmul is matrix multiplication;

[0114] The feature map enhanced by the improved CBAM attention module is input into the detection head to calculate the loss. In the detection head, the classification loss is calculated using the Binary cross-entropy loss function and the manually labeled true category value. The bounding box loss is calculated using the Distribution Focal loss function and the Ciou loss function and the manually labeled true normalized bounding box coordinate value. The classification loss and bounding box loss are weighted averaged to obtain the network loss. The specific situation is shown in formulas (18), (19), and (20):

[0115] L cls =f bce (p pred ,p gt ) (18)

[0116] L bbox =f ciou (b pred ,b gt )+ω dfl f dfl (b pred ,b gt) (19)

[0117] L=ω cls L cls +ω bbox L bbox (20)

[0118] Where, L cls represents the classification loss, L bbox represents the bounding box loss, L represents the total loss, and p pred Indicates the probability that the network predicts the target to be a specified category, p gt Indicates the true probability that the target is a specified category, b pred represents the bounding box predicted by the network, b gt represents the true bounding box, ω dfl Represents the weight corresponding to the Distribution focal loss, ω cls Represents the weight corresponding to the classification loss, ω bbox Represents the weight corresponding to the bounding box loss. In this method, ω dfl =0.375,ω cls =0.5,ω bbox =7.5,f bce Represents the Binary cross-entropy loss function, f ciou represents the Ciou loss function, f dfl Denotes the Distribution focal loss function, which is calculated as shown in formulas (21), (22), and (23):

[0119] f bce (p pred ,p gt )=-(p gt log(p pred )+(1-p gt )log(1-p pred )) (twenty one)

[0120]

[0121]

[0122] Where, IOU represents b pred and b gt The intersection-over-union ratio, ρ represents b pred and b gt The Euclidean distance of b, c represents pred and b gt The diagonal length of the minimum bounding rectangle, γ gt Indicates b gt The aspect ratio, γpred Indicates b pred The aspect ratio of the bounding box is τ, which means rounding up the coordinates of the four points of the bounding box, and ψ which means rounding down the coordinates of the four points of the bounding box;

[0123] After calculating the network loss, the gradient descent strategy and loss calculation back propagation are used to optimize and update the network parameter weights. The total number of iterations of the network training process is 500. The validation set is used to verify the performance of the network every 10 iterations. The termination condition of the training is that the performance indicators of the network in the validation set no longer change significantly and the total loss of the network no longer decreases.

[0124] 3) The test set is sent to the network with the best performance to obtain the classification value predicted by the network and the coordinates of the predicted bounding box. When the recognized classification value is 0, the recognition result is the background, and it is judged to be a normal car door surface. When the recognized classification value is 1, the recognition result is a scratch, and it is judged that there is a scratch defect in the car door surface image. When the recognized classification value is 2, the recognition result is a pinhole, and it is judged that there is a pinhole defect in the car door surface image. When the recognized classification value is 3, the recognition result is sheet metal, and it is judged that there is a sheet metal defect in the car door surface image. Among them, the precision and recall rates of various defects during the test process are recorded, and the evaluation index of the best performance of the network is obtained by the highest precision and recall rate.

[0125] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A door surface defect detection method based on an improved YOLOv8 network is characterized by: The improved YOLOv8 network includes two improvements. The first part is to improve the residual module of the backbone network: the two serial 3×3 convolutional layers are replaced by a 3×3 depthwise separable convolutional layer and a 3×3 Ghost-Dilated convolutional layer to construct parallel feature extraction. Different features are extracted through parallel convolution operations and spliced, thereby reducing the number of parameters and computational complexity of the backbone network. The second part is to improve the fusion method of different features of the neck network. Instead of fusing two adjacent features of different scales, the features of all scales are scaled into a feature map of a unified scale. The mean of the feature maps of different scales is then calculated to obtain a mean feature map. The obtained mean feature map is fused with the features of different scales, and the improved CBAM attention module is used to aggregate the information of the fused feature map. The improvement of the CBAM attention module is to replace the spatial attention module with a context aggregation module. The context aggregation module is a self-attention module used to enhance the global information of the fused features, thereby reducing the impact of imbalance on network accuracy during the training process of door surface defect detection. The specific implementation of the vehicle door surface defect detection method includes the following steps: 1) Collect door surface defect images, build a custom door surface defect dataset, and divide it into training set, validation set, and test set; 2) The training set is fed into the improved YOLOv8 network in batches for training. After the images in the training set are feature extracted by the backbone network, the feature layers of different scales are fused and the extracted features are enhanced through the neck network. The network loss is obtained through the detection head, and the network parameters and weights are updated based on the gradient descent strategy and loss calculation backpropagation. In addition, the network performance is verified on the validation set after each preset number of network training. When the evaluation index on the validation set reaches the optimal value and the network loss value no longer decreases, the network with the best performance after training and verification is obtained. 3) The test set is fed into the best performing network obtained in step 2) to obtain the predicted classification value and the coordinates of the predicted bounding box, where different classification values ​​correspond to different defects.

2. The vehicle door surface defect detection method based on the improved YOLOv8 network according to claim 1 is characterized in that: In step 1), before photographing the door surface, the robot arm is first controlled to the designated photographing area, and the shape of the stripes in the stripe light source is adjusted according to different simulated production conditions. After the image displayed by the host computer stabilizes, the host computer controls the camera to photograph the door surface defect image, and then adjusts the position of the robot arm to traverse the entire door to complete the acquisition of the door surface defect image; After the acquisition is completed, the image is preprocessed. The preprocessing operation flow includes: first, bilinear interpolation is used to scale the original image, unify the size to a resolution of 640×640, adjust the saturation, hue and exposure of the image, and then perform Gaussian filtering on the transformed image to reduce the impact of noise in the image on the detection accuracy; the preprocessed image is used to construct a door surface defect dataset, which includes normal door surface images, pinhole defect images, scratch defect images and sheet metal defect images. The Mosaic method is used to perform data enhancement on pinhole defect images and sheet metal defect images. The Mosaic method first randomly selects a position as the position of the center image. The position is randomly selected from an area in the input image, and the selected center image is placed at the center of the input image. The boundary image is placed on the four boundaries of the input image after scaling, translation and flipping operations. Finally, the target bounding box in the boundary image is adjusted according to the position and scaling ratio of the boundary image in the input image. The constructed door surface defect dataset is divided into training set, validation set and test set according to the set ratio.

3. The vehicle door surface defect detection method based on the improved YOLOv8 network according to claim 2 is characterized in that: In step 2), the improved YOLOv8 network includes three parts: backbone network, neck network and detection head. The details are as follows: The first layer of the backbone network is a 3×3 convolution operation, which is used to adjust the number of channels of the input image and downsample the input size; The second and fifth layers are both constructed with a 3×3 convolution operation followed by an improved residual module operation repeated three times. The third and fourth layers are both constructed with a 3×3 convolution operation followed by an improved residual module operation repeated six times. The improved residual module extracts features in five steps, as follows: The first step is a 3×3 convolution operation that adjusts the channel dimension of the original input feature map to twice its original size, obtaining a new input feature map I. The second step is to divide the input feature map I into two sub-feature maps I1 and I2 according to the channel dimension; The third step is to perform a 3×3 depth-wise separable convolution operation on the sub-feature map I1 to form a new feature map I'1. The process is as follows: I'1=I1*DW3 (1) Where I'1 represents the feature map output by the 3×3 depthwise separable convolution operation, DW3 represents the 3×3 depthwise separable convolution operation. The depthwise separable convolution operation splits the standard convolution operation into two steps: depthwise convolution operation and pointwise convolution operation. The depthwise convolution operation independently performs convolution operation on each channel of the input feature map, and the pointwise convolution operation uses a 1×1 convolution kernel to perform convolution operation on the output of the depthwise convolution operation. The process is as follows: X'1=X1*K3*K1 (2) In the formula, X'1 represents the feature of one channel of I'1, X1 represents the feature of one channel of I1, K3 represents a 3×3 convolution operation, and K1 represents a 1×1 convolution operation; The fourth step is to perform a 3×3 Ghost-Dilated convolution operation on the sub-feature map I2 to form a new feature map I'2. The Ghost-Dilated convolution operation splits the convolution operation into three steps: main branch convolution operation, pseudo branch convolution operation and feature fusion operation. Among them, the main branch convolution operation operates on 75% of the channels of the input feature map and uses a 3×3 convolution operation to process the input features. The pseudo branch convolution operation operates on 25% of the channels of the input feature map and uses a 7×7 hole convolution operation to process the input features. Finally, the output results of the two branches are subjected to a feature splicing operation. The process is as follows: Where I'2 represents the feature map output by the 3×3 Ghost-Dilated convolution operation, The characteristic of the 75% channel representing I2, Represents the characteristics of the 25% channel of I2, represents a 7×7 dilated convolution operation, and C represents a feature concatenation operation in the channel dimension; The fifth step is to obtain a new feature map I3 by performing feature concatenation operation on the feature map I'1 and the feature map I'2, and to add the feature map I3 to the feature map I to form a new feature map I4, thereby fusing the information between the feature maps obtained by different types of convolution methods and enriching the features between different semantic information of the image. The process is as follows: I3=C(I'1,I'2) (4) I4=I+I3 (5) Where, I3 represents the feature map obtained by concatenating the feature map I'1 and the feature map I'2, and I4 represents the feature map obtained by adding the feature map I and the feature map I3; The neck network uses an improved feature fusion method to fuse the features F3, F4, and F5 output by the three to five layers of the backbone network. This improved feature fusion method is divided into four steps to fuse features. The details are as follows: The first step is to scale F3 and F5 to the same scale as F4. The process is as follows: F'3=D(F3) (6) F'5=U(F5) (7) Where F'3 represents the scaled output of F3, F'5 represents the scaled output of F5, D represents the maximum pooling operation, and U represents the bilinear difference operation; The second step is to perform weighted averaging using F'3, F4, F'5 and the preset hyperparameters α, β, and γ. The process is as follows: F avg =αF'3+βF4+γF'5 (8) Where, F avg is the average feature output by the weighted average operation; The third step is to avg The feature fusion process with F3, F4, and F5 is as follows: <h2 style=";text-align:left;direction:ltr">P3=C(F3,F<h2 style=";text-align:left;direction:ltr"> avg <h2 style=";text-align:left;direction:ltr"> *K1) (9) P4=C(F4,F avg *K1) (10) P5=C(F5,F avg *K1) (11) Where, P3 is F3 and F avg The output feature after fusion, P4 is F4 and F avg The output features after fusion, P5 is F5 and F avg The output features after fusion; The fused features P3, P4, and P5 are enhanced through the improved CBAM attention module. The improved CBAM attention module is divided into a channel attention module and a context aggregation module. The channel attention module calculates the weight vector λ i Applied to the input features, the important channels are highlighted and the unimportant channels are suppressed by element-by-element multiplication. The specific situation is as shown in formulas (12) and (13): l i =σ(f c (f gap (P i )))i=3,4,5 (12) P' i =λ i P i i=3,4,5 (13) Where P' i represents the output feature after the channel attention module is enhanced, σ represents the Sigmoid function, f c represents the fully connected layer, and f gap Global average pooling operation, i represents the i-th layer architecture, P i represents the output features of the i-th layer architecture, λ i Indicates P i The corresponding weight vector; Then, P' i As input features, it is input into the context aggregation module, which is fed into the query feature map P' iq , key feature graph P' ik , value characteristic graph P' iv Attention calculation is performed on each position in the entire feature map, as shown in formulas (14), (15), (16), and (17): P' iq =P' i *K1 i=3,4,5 (14) P' iv =σ(P' i *K1) and=3,4,5 (15) P i ”=f matmul (P' iv ,P' iq ·P' ik ) i=3,4,5 (17) Where, P i " represents the features enhanced by the context aggregation module, represents the Softmax function, f matmul is matrix multiplication; The feature map enhanced by the improved CBAM attention module is input into the detection head to calculate the loss. In the detection head, the classification loss is calculated using the Binary cross-entropy loss function and the manually labeled true category value. The bounding box loss is calculated using the Distribution Focal loss function and the Ciou loss function and the manually labeled true normalized bounding box coordinate value. The classification loss and bounding box loss are weighted averaged to obtain the network loss. The specific situation is shown in formulas (18), (19), and (20): L cls =f bce (p pred ,p gt ) (18) L bbox =f ciou (b pred ,b gt )+ω dfl f dfl (b pred ,b gt ) (19) L=ω cls L cls +oh bbox L bbox (20) Where, L cls represents the classification loss, L bbox represents the bounding box loss, L represents the total loss, and p pred Indicates the probability that the network predicts the target to be a specified category, p gt Indicates the true probability that the target is a specified category, b pred represents the bounding box predicted by the network, b gt represents the true bounding box, ω dfl Represents the weight corresponding to the Distribution focal loss, ω cls Represents the weight corresponding to the classification loss, ω bbox represents the weight corresponding to the bounding box loss, f bce Represents the Binary cross-entropy loss function, f ciou represents the Ciou loss function, f dfl Denotes the Distribution focal loss function, which is calculated as shown in formulas (21), (22), and (23): f bce (p pred ,p gt )=-(p gt log(p pred )+(1-p gt )log(1-p pred )) (21) Where, IOU represents b pred and b gt The intersection-over-union ratio, ρ represents b pred and b gt The Euclidean distance of b, c represents pred and b gt The diagonal length of the minimum bounding rectangle, γ gt Indicates b gt The aspect ratio, γ pred Indicates b pred The aspect ratio of the bounding box is τ, which means rounding up the coordinates of the four points of the bounding box, and ψ which means rounding down the coordinates of the four points of the bounding box; After calculating the network loss, the gradient descent strategy and loss calculation back propagation are used to optimize and update the network parameter weights; the total number of iterations of the network training process is 500, and the validation set is used to verify the network performance every 10 iterations.

4. The vehicle door surface defect detection method based on the improved YOLOv8 network according to claim 3 is characterized in that: In step 3), the test set is sent to the network with the best performance to obtain the classification value predicted by the network and the coordinates of the predicted bounding box. When the recognized classification value is 0, the recognition result is the background, and it is determined to be a normal car door surface. When the recognized classification value is 1, the recognition result is a scratch, and it is determined that there is a scratch defect in the car door surface image. When the recognized classification value is 2, the recognition result is a pinhole, and it is determined that there is a pinhole defect in the car door surface image. When the recognized classification value is 3, the recognition result is sheet metal, and it is determined that there is a sheet metal defect in the car door surface image. The precision and recall rates of various defects during the test are recorded, and the evaluation index of the best performance of the network is obtained through the highest precision and recall rate.

Images