An Integrated Approach for Paint Defect Detection and Segmentation Based on the YOLOv5 Framework

Abstract

This invention discloses an integrated method for vehicle paint defect detection and segmentation based on the YOLOv5 framework, comprising: acquiring a target vehicle paint image; inputting the target vehicle paint image into a trained vehicle paint defect detection and segmentation model, and outputting a vehicle paint defect detection image and a segmentation image corresponding to the target vehicle paint image; the vehicle paint defect detection and segmentation model is constructed using the YOLOv5 framework and includes segmentation and detection branches. This method can improve detection and segmentation efficiency while enhancing detection and segmentation accuracy when detecting and segmenting vehicle paint.

Description

Technical Field

[0001] This invention belongs to the fields of machine learning and digital image processing technology, and in particular to an integrated method for detecting and segmenting automotive paint defects based on the YOLOv5 framework. Background Technology

[0002] Vehicle paint serves important functions such as aesthetics, corrosion resistance, and sun protection. The surface quality of the paint has become a crucial indicator of a vehicle's performance, and the role of paint surface quality control in industrial production is increasingly significant. Traditional manufacturing processes rely heavily on visual inspection to detect surface defects such as scratches, cracks, and chips generated during production. This method suffers from high rates of missed defects, low production efficiency, and an inability to provide accurate data. Furthermore, it involves high labor intensity, poses significant health risks to workers, and is difficult and costly to recruit.

[0003] Machine vision-based automatic surface defect detection, as a rapidly developing new detection technology, boasts advantages such as high speed and high quality. It can overcome the shortcomings of current manual inspection methods, such as excessive time consumption and low first-pass detection rates, while also reducing labor costs. Current industrial applications use methods like tunneling scanning and phase deflection to obtain images of paint defects. However, due to the design of image scanning systems and advancements in paint processing technology, in an image containing defects, the paint defects often occupy only a small portion of the image, while the background is relatively uniform. Furthermore, the overall data shows a significant difference in the number of different types of paint defects, resulting in very few images containing paint defects. Paint defects are typical weak targets, characterized by their small number, unbalanced categories, weak texture, and low contrast.

[0004] Object detection aims to detect objects in an input image, label their locations, and determine their classification. Semantic segmentation, on the other hand, classifies objects pixel-by-pixel in the input image, identifying them at the pixel level. Currently, most deep learning-based defect detection methods merely outline defects (object detection), resulting in less than ideal defect detection performance. In fact, extracting defect regions at the pixel level (semantic segmentation) not only reveals their location but also better reflects real-world application needs.

[0005] The YOLOv5 model was publicly released by Ultralytics on June 9, 2020. Based on the YOLOv3 model, the YOLOv5 model includes four sub-models: YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. The YOLOv5 model consists of a backbone network, a neck layer, and a head layer. The backbone network aggregates image features at different fine-grained levels to form a convolutional neural network. The neck layer is a series of network layers that mix and combine image features and pass these features to the prediction layer. The head layer predicts the image features, generates bounding boxes, and predicts the category.

[0006] Copy-paste data augmentation methods obtain rich and novel training data by pasting different objects of varying sizes onto new background images. The main idea is a combination of hybrid pasting and large-scale jittering.

[0007] Therefore, how to simultaneously perform vehicle paint defect detection and segmentation based on the YOLOv5 framework, while also addressing or mitigating the negative impact of insufficient data, to better meet the needs of practical application scenarios, has become a key research issue. Summary of the Invention

[0008] In view of the above problems, the present invention provides an integrated method for vehicle paint defect detection and segmentation based on the YOLOv5 framework that at least solves some of the above technical problems. This method can effectively complete vehicle paint defect detection and segmentation.

[0009] This invention provides an integrated method for vehicle paint defect detection and segmentation based on the YOLOv5 framework, including:

[0010] S1. Obtain the target vehicle paint image;

[0011] S2. Input the target paint image into the trained paint defect detection and segmentation model, and output the paint defect detection image and segmentation image corresponding to the target paint image; the paint defect detection and segmentation model adopts the YOLOv5 framework and includes a segmentation branch and a detection branch.

[0012] Furthermore, in S2, the training process of the vehicle paint defect detection and segmentation model is as follows:

[0013] Step 1: Obtain a large number of vehicle paint defect images with detection and segmentation annotations as a training dataset; and preprocess the training dataset.

[0014] Step 2: Perform data augmentation on the preprocessed training dataset using the copy-paste method constrained by the k-means clustering method;

[0015] Step 3: Add a split header to the YOLOv5 framework to obtain split branches;

[0016] Step 4: Redesign the detection head of the YOLOv5 framework to obtain the detection branches;

[0017] Step 5: Based on the training dataset after data augmentation, train the segmentation branch and the detection branch in turn until convergence to obtain the trained paint defect detection and segmentation model.

[0018] Furthermore, in step one, the preprocessing includes: dividing the training dataset using cross-validation to obtain K non-overlapping subsets.

[0019] Furthermore, step two specifically includes: using the k-means clustering method to divide the images in the preprocessed training dataset into bright regions and dark regions; and using the copy-paste method in the bright regions and dark regions respectively to perform data augmentation processing on the preprocessed training dataset.

[0020] Furthermore, step three specifically includes:

[0021] The YOLOv5 framework uses an SPPF+FPN structure at the neck; at the head of the YOLOv5 framework, the lower layer of the FPN network is fed into the segmentation head to obtain the segmentation branches.

[0022] Features are extracted from the backbone network of the YOLOv5 framework;

[0023] The features are fused using an SPPF network to obtain features at different semantic scales;

[0024] The features at different semantic scales are fed into the top-down upsampling process of the FPN network. The FPN network splices and fuses the features at different semantic scales, and the bottom layer of the FPN is fed into the segmentation branch. After three upsampling operations, the segmentation result is obtained.

[0025] Furthermore, step four specifically includes:

[0026] Add a decoupling header to the YOLOv5 framework;

[0027] Introducing class weights into the YOLOv5 framework;

[0028] Add a small-eye detection layer to the detection layer of the YOLOv5 framework;

[0029] Add a new set of anchor values ​​to the detection layer of the YOLOv5 framework.

[0030] Furthermore, the new anchor value is smaller than the original anchor value in the YOLOv5 framework.

[0031] Furthermore, in step five, the segmentation branch and the detection branch are trained sequentially in turn, specifically including: performing forward propagation of the detection branch and the segmentation branch respectively; and updating the weights in the detection branch and the segmentation branch in turn during the back propagation process.

[0032] Furthermore, in step five, the model is trained and validated K times for the segmentation branch and the detection branch; each time, a subset of the dataset is used to validate the model, and another K-1 subsets of the dataset are used to train the model; finally, the average of the training error and validation error of these K times is calculated.

[0033] Compared with existing technologies, the integrated method for vehicle paint defect detection and segmentation based on the YOLOv5 framework described in this invention has the following beneficial effects:

[0034] The paint defect detection and segmentation model of this invention adopts the YOLOv5 framework and includes segmentation and detection branches. When detecting and segmenting paint defects, it utilizes an FPN network to extract and fuse features at different levels, improving both detection and segmentation efficiency and accuracy. This allows for simultaneous paint defect detection and segmentation, better meeting the needs of practical applications. Segmentation generates a mask image of the defect target; detection yields the type and location of the paint defect, while segmentation determines its shape and size.

[0035] In this invention, during the bottom-up process of the FPN network, the deep network focuses on semantic information, while the shallow network focuses on detailed information. Then, in the top-down process, the detection branch utilizes the information from the deep layers to complete the detection task, addressing the questions of "what is the paint defect?" and "where is the paint defect?". The subsequent upsampling essentially utilizes the detailed information from the bottom-up process to obtain the size and shape of the paint defect. If a direct segmentation network (such as Unet) is used, its structure will inevitably be shallow due to the limited dataset size, resulting in insufficient feature extraction and a high false negative rate. Therefore, compared to direct segmentation algorithms, this invention uses detection as an aid to segmentation, achieving better results. Furthermore, the method of further integrating detailed information in the detection branch combines semantic and detailed information, ultimately yielding better results.

[0036] This invention incorporates an additional small target detection layer and anchor for small targets, making the model more adaptable to situations with imbalanced positive and negative data, small targets, and a simple background.

[0037] This invention addresses the problem of insufficient data by employing the k-means clustering method to constrain the copy-paste process, taking into account the characteristics of the data. This method effectively solves the problem of insufficient data and reduces the negative impact of direct copy-paste.

[0038] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0039] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0040] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0041] Figure 1 This is a schematic diagram of the framework of the vehicle paint defect detection and segmentation model provided in an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of a paint defect image provided in an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of the FPN structure provided in an embodiment of the present invention.

[0044] Figure 4 This is a schematic diagram of the decoupling head structure provided in an embodiment of the present invention.

[0045] Figure 5 This is a schematic diagram of the output results provided in an embodiment of the present invention. Detailed Implementation

[0046] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0047] This invention provides an integrated method for detecting and segmenting automotive paint defects based on the YOLOv5 framework, specifically including the following steps:

[0048] S1. Obtain the target vehicle paint image;

[0049] S2. Input the target paint image into the trained paint defect detection and segmentation model, and output the paint defect detection image and segmentation image corresponding to the target paint image; the paint defect detection and segmentation model adopts the YOLOv5 framework and includes a segmentation branch and a detection branch.

[0050] In S2 above, the training method for the paint defect detection and segmentation model is as follows:

[0051] Step 1: Obtain a large number of vehicle paint defect images with detection and segmentation annotations as a training dataset; and preprocess the training dataset.

[0052] Step 2: Perform data augmentation on the preprocessed training dataset using the copy-paste method constrained by the k-means clustering method;

[0053] Step 3: Add a split header to the YOLOv5 framework to obtain split branches;

[0054] Step 4: Redesign the detection head of the YOLOv5 framework to obtain the detection branches;

[0055] Step 5: Based on the training dataset after data augmentation, train the segmentation branch and the detection branch until convergence, and obtain the trained car paint defect detection and segmentation model.

[0056] See the schematic diagram of the framework for the vehicle paint defect detection and segmentation model. Figure 1 As shown.

[0057] The following is a detailed explanation of steps one through five.

[0058] In step one above, to simultaneously complete the detection and segmentation tasks, in this embodiment of the invention, more than 200 960x480 images of vehicle paint defects with detection and segmentation labels were prepared; each image of vehicle paint defects contains a varying number and size of defects, as detailed below. Figure 2 As shown, these images of paint defects are used as the training dataset; cross-validation is then used to divide the training dataset into K non-overlapping subsets.

[0059] In step two above, due to the small number of training datasets, small target size, and large brightness variations, the traditional copy-paste method was first used in this embodiment to enrich the dataset and mitigate the impact of insufficient data. The copy-paste method is essentially a combination of mixed pasting and large-scale jittering. Mixed pasting involves copying the object from the original image and then pasting it into a new image; large-scale jittering involves significant size changes in the pasted object. This method copies different objects of varying sizes into a new background image, creating rich and novel training data. However, due to the undulating surface of the car, the reflection components in the image vary considerably, resulting in a clear regional distribution of brightness. This causes the copied and pasted target to carry the domain information of the original target. Therefore, directly using the copy-paste method is ineffective.

[0060] K-means clustering is a classic clustering method. This method first randomly selects K centroids, then assigns all data points to the nearest centroid based on distance, resulting in K clusters. The mean of these clusters is then calculated to obtain K new centroids. These new centroids are further divided into K new clusters, and this process is repeated until the algorithm converges. This algorithm divides the dataset into K clusters, ensuring small intra-cluster distances and large inter-cluster distances. Therefore, in this embodiment of the invention, the K-means clustering method is chosen to constrain the copy-paste process; the K-means clustering method is first used to divide the image into regions based on brightness, and the silhouette coefficient is used as an evaluation metric.

[0061] The profile coefficient is expressed as

[0062]

[0063] The formula for calculating the average distance is:

[0064]

[0065] Where a(i) represents the average distance of other samples within the cluster to which sample i belongs. If there is only one sample i in the cluster, then let S(i) = 0. In the formula for average distance, i represents the i-th sample; j represents other sample points in the same class as sample i; n is the number of sample points in the same class as i; b(i) represents the minimum average distance between the i-th sample and samples from other clusters. At this time, j represents a sample point in a class that is different from i; n is the number of sample points in that class. The effect is better when S approaches 1.

[0066] After analyzing and testing the data, this embodiment of the invention ultimately decided to use the k-means clustering method to divide the image into four regions (bright, relatively bright, relatively dark, and dark). The brightness of the same region is similar, and different regions have obvious boundaries. This embodiment of the invention uses the copy-paste method in the same region, which can effectively reduce the negative impact of direct copy-paste and effectively make up for the problems of small size and small number of defects in the dataset.

[0067] In step three above, the YOLOv5 model was originally used for detection tasks. In this embodiment of the invention, to complete the segmentation task, a segmentation head is added to the YOLOv5 framework for semantic segmentation. The overall framework of the model is as follows: Figure 1 As shown. This embodiment of the invention is based on the YOLOv5 branch YOLOv5s, and uses an SPPF+FPN structure in the neck of the YOLOv5 framework to process the features extracted by the backbone network; wherein, SPPF generates and fuses features at different semantic scales, FPN fuses features at different semantic levels, and uses a concatenation method to fuse features at different scales. Figure 1 As shown, after the YOLOv5 backbone network extracts features, this embodiment of the invention first uses SPPF to fuse the feature maps extracted by the backbone network, and then feeds the result into the top-down upsampling process of the FPN network. Finally, at the head of the YOLOv5 framework, the bottom layer of the FPN is fed into the segmentation head to obtain a segmentation branch with a size of (W / 8, H / 8, 256). After three upsampling processes, the output feature map is restored to a size of (W, H, 2), which represents the probability of each pixel in the input image for defects and background, thereby achieving the segmentation goal.

[0068] Next, we will explain SPPF+FPN in detail in this step:

[0069] 1. SPPF (Spatial Pyramid Pooling-Fast):

[0070] Spatial Pyramid Pooling (SPP) can transform feature maps of arbitrary size into feature vectors of fixed size. SPP removes the requirement of fixed input for CNNs, and its operation is similar to deep information "fusion." This deep information fusion is similar to the hierarchical information processing method of the human brain; when people see an object, they initially perceive it as a whole rather than a segmented object; by fusing previously processed information, the brain recognizes objects of various shapes at a deeper level. The advantages of SPP are that it produces fixed output features under inputs of arbitrary size and scale; and it is robust to object deformation due to the use of multi-level spatial bins (pooling at multiple sizes).

[0071] SPPF (Spatial Pyramid Pooling-Fast) is an improved version of SPP. Its structure is as follows: The input feature map undergoes a CBL operation, where CBL is a 1x1 convolution + Batch Normalization (BN) + SiLu activation function. Then, three max pooling operations are performed with kernels of 5, 9, and 13 respectively. The CBL-processed feature map is concatenated with the results of the three pooling operations, and then another CBL operation is performed before output. SPPF is faster than SPP.

[0072] 2. FPN (Feature Pyramid Networks):

[0073] In convolutional networks, deep networks readily respond to semantic features, while shallow networks readily respond to image features. While high-level networks can respond to semantic features, their small feature map size limits their geometric information, hindering object detection. Conversely, shallow networks contain more geometric information but lack semantic features, making them less suitable for image classification. This problem is particularly pronounced in small object detection. Therefore, a network is needed that can combine deep and shallow features, simultaneously satisfying the needs of object detection and image classification.

[0074] Feature Pyramid Networks (FPN) is a top-down feature fusion method, but it's also a multi-scale object detection algorithm, meaning it doesn't have just one feature prediction layer. While some algorithms also use multi-scale feature fusion for object detection, they often only utilize features at one scale after fusion. This approach, while capturing semantic information from top-level features and detailed information from lower-level features, can introduce biases during feature deconvolution and other processes, negatively impacting detection accuracy. FPN addresses this issue by performing predictions on fused features at multiple different scales, maximizing detection accuracy. FPN's model training differs from the traditional Faster R-CNN method because it has multiple prediction layers. Similar to SSD, the earlier features in FPN have higher resolution, and the purpose of this embodiment is to use high-resolution features to predict small targets. Each fused feature in the FPN is connected to an RPN (Region Proposal Network) to generate candidate boxes, and then all the generated candidate boxes are aggregated together. Because in this embodiment of the invention, the goal is to predict on a higher resolution convolutional feature map with a smaller target, these candidate boxes are assigned, and the assignment principle is as follows:

[0075]

[0076] Where k0 is a constant representing the prediction layer to which the candidate box is assigned; w and h represent the width and height of the candidate box; [] represents the next lower bound of the value. For example, there are four prediction layers: 2, 3, 4, and 5, representing four fusion feature layers with resolutions from largest to smallest. When k0 = 4, a candidate box with a width-to-height product of 112*2 will be assigned to prediction layer k = 3 (k is 5 when it is greater than 5; k is 2 when it is less than 2) for subsequent detection. As can be seen from the formula, the larger the area of ​​the candidate box, the larger the prediction layer number it is assigned to; larger targets will be assigned to a lower resolution prediction layer for prediction. After these candidate boxes are assigned, they are fed into the corresponding RoI pooling layer. The outputs of the RoI pooling layer are concatenated together and then passed through two fully connected layers for target classification and target location regression.

[0077] FPN uses feature maps of different resolutions to perceive objects of different sizes, and through continuous upsampling and cross-layer fusion mechanisms, the output features combine low-level visual information with high-level semantic information. Low-level feature maps lack sufficient semantic richness and cannot be directly used for classification, while deeper features are more reliable. By combining lateral connections with top-down connections, feature maps of different resolutions can be obtained, all of which contain the semantic information of the original deepest feature map.

[0078] The idea behind Feature Pyramid Networks (FPN) originates from multi-scale recognition in traditional algorithms. Specifically, it scales the original image to different scales; the scaled-down image is used for global features, while the scaled-up image is used for detailed features. Deeper feature maps in a deep learning network possess more global and abstract features. FPN addresses the issue of combining shallow and deep features by using horizontal connections and vertical addition. The overall structure adopts a top-down information structure; see [link to details]. Figure 3 As shown: In the diagram, each square represents a feature map, with squares from smallest to largest indicating feature maps from deep to shallow. In the newly generated feature maps, the shallowest layer contains features from all the layers above. Deeper layers have less semantic information but more accurate target location; shallower layers have richer semantic information but coarser target location. The top of the diagram shows a network structure with skip connections, where prediction is performed at the finest level (the last layer from top to bottom). Simply put, features are upsampled multiple times and fused together in the final step, using the features generated in the last step for prediction. The bottom of the diagram shows a similar network structure, but prediction is performed independently in each layer.

[0079] The FPN connection structure employs 1x1 convolutional layers and a 2x upsampling mechanism. Upsampling is the reverse process of convolution, also known as deconvolution. Convolution operations continuously shrink or keep the feature map size constant (1x1 convolution), while deconvolution operations continuously increase the feature map size. Deconvolution is used here to enlarge deeper feature maps compared to shallower ones. Figure 1After resizing, element-wise addition can be performed. The 1x1 convolutional structure is designed to perform a size-invariant spatial transformation on the original feature map, making the structure more robust. During the forward pass, the size of the feature map changes after passing through some layers, but remains unchanged after passing through others. The FPN authors grouped layers that do not change the feature map size into a stage, so each extracted feature is the output of the last layer of each stage, thus forming a feature pyramid. The output of the last layer of each stage is the feature map of the pyramid. The top-down process uses upsampling, while the lateral connections fuse the upsampled result with the bottom-up generated feature map of the same size. After fusion, a 3x3 convolutional kernel is used to convolve each fused result to eliminate the aliasing effect of upsampling.

[0080] In step four above, since the targets processed in this embodiment of the invention are characterized by small number, small size, and weak texture, this embodiment of the invention requires special design of the detection branch of the YOLOv5 framework, specifically including:

[0081] 1. Add a decoupling header to improve accuracy on small datasets:

[0082] Object detection's localization and classification tasks differ in their focus and interest. Classification focuses on which existing class the extracted features are most similar to, while localization focuses on the positional coordinates of the ground truth box (GT Box) to refine bounding box parameters. The conflict between classification and regression tasks is a well-known problem in object detection. Therefore, decoupling heads for classification and localization are widely used in most one-stage and two-stage detectors. However, although the YOLO series backbone and feature pyramids have evolved, their detection heads remain coupled, which can degrade performance. Structurally, fully connected heads are better suited for classification tasks, while convolutional heads are better suited for localization tasks. Instance segmentation requires outputting a classification confidence map and an instance mask map; object detection requires outputting the object's classification and bounding box location. Using two different heads to decouple the outputs of classification and location constitutes a decoupling head.

[0083] YOLOX proposes a decoupling header structure, see [link / reference] Figure 4As shown, YOLOX's decoupling head structure separates the convolutional head and the fully connected head into two separate paths. The decoupling head accepts the feature map extracted by the FPN network as input, uses a 1x1 convolutional layer to reduce the number of channels in the feature map, and outputs H*W*C (W and H represent the image width and height, and C represents the number of image channels). It then uses two parallel 3x3 convolutional layers to complete the regression and classification tasks (outputs are H*W*4 and H*W*1, respectively).

[0084] The YOLOv5 framework incorporates an anchor matching process in its head. The original YOLOv5 framework expands the number of channels in the head and neck feature maps of different scales using 1×1 convolutions. The expanded feature channel count is (number of categories + 5) × the number of anchors on each detection layer. Here, 5 represents the x-coordinate, y-coordinate, width, height, and confidence score of the predicted bounding box's center point. The confidence score represents the likelihood of the predicted bounding box's presence, ranging from (0, 1), with higher values ​​indicating a greater probability of a target within the bounding box. Each grid cell in each detection layer of the YOLOv5 framework pre-sets multiple anchor prior boxes. These prior boxes need to be filtered, classifying them as positive and negative samples. The three detection layers in the head correspond to three different sized feature maps obtained in the neck. Each grid cell in the feature map pre-sets three anchors with different aspect ratios, storing all positional and classification information based on the anchor prior boxes along the channel dimension of the feature map for target prediction and regression.

[0085] The YOLOv5 bounding box regression calculation formula is shown below:

[0086] b x =(2*σ(t) x )-0.5)+c x

[0087] b y =(2*σ(t) y )-0.5)+c y

[0088] b w =p w *(2*σ(t w )) 2

[0089] b h =p h *(2*σ(t h )) 2

[0090] Among them, b x b y b wand b h These represent the center point coordinates, width, and height of the prediction box, respectively; (C x C y (t) represents the coordinates of the top-left corner of the grid where the center point of the prediction box is located; x , t y (t) represents the offset of the center point of the prediction box relative to the coordinates of the top-left corner of the grid; w , t h (p) represents the scaling ratio of the predicted bounding box's width and height relative to the anchor's width and height. w p h The values ​​() represent the width and height of the prior bounding box anchor. To constrain the center point of the predicted bounding box to the current mesh, the Sigmoid function is used to handle the offset, keeping the predicted offset value within the range of (0, 1). Thus, according to the target bounding box regression calculation formula, the offset of the predicted bounding box center point coordinates remains within the range of (-0.5, 1.5). The width and height of the predicted bounding box are scaled to the anchor within the range of (0, 4).

[0091] YOLOv5 employs a matching strategy based on aspect ratio. For each Ground Truth (human-annotated ground truth bounding box), the width-to-width ratio (w1 / w2, w2 / w1) and height-to-height ratio (h1 / h2, h2 / h1) of it and nine different anchors are calculated. The maximum value among the width ratios (w1 / w2, w2 / w1) and height ratios (h1 / h2, h2 / h1) of the Ground Truth and the anchor is found and used as the ratio between the Ground Truth and the anchor. If the ratio r^max of the Ground Truth and the anchor is less than the set ratio threshold (the default hyperparameter is anchor_t = 4.0), then this anchor is responsible for predicting this Ground Truth, and the predicted bounding box regressed by this anchor is called a positive sample. All other predicted bounding boxes are negative samples.

[0092] Based on this, referencing the decoupling head structure proposed by YOLOX, this embodiment of the invention introduces a decoupling head structure into the YOLOv5 framework. The design of the decoupling head in this embodiment retains the anchor detection method in the YOLOv5 framework, that is, anchor detection is performed within the YOLOX decoupling head structure. Finally, the output of each of the three parts of the decoupling head includes an additional anchor term (i.e., H*W*anchor*C, H*W*anchor*4, H*W*anchor*1). Considering the different annotations for classification and localization, the decoupling head structure uses different branches for computation, accelerating network convergence and improving accuracy.

[0093] 2. Use class weights to balance highly imbalanced samples:

[0094] The training dataset used in this embodiment of the invention consists of highly imbalanced samples. Most machine learning algorithms lack effectiveness with biased class data; however, this embodiment of the invention modifies existing training algorithms to account for skewed class distributions. This can be achieved by assigning different weights to the majority and minority classes. During training, these weight differences affect class classification. The overall goal is to penalize misclassification of the minority classes by setting higher class weights while lowering the weights for the majority class.

[0095] The basic idea of ​​the class weighting method is to assign greater weight to the minority class in the algorithm's cost function, allowing it to provide a higher penalty and enabling the algorithm to focus on reducing the error of the minority class. For a given class, the weight calculation formula is: in:

[0096] w j The weights of each class are represented by j (where j represents the class); n_samples represents the total number of samples or rows in the dataset; n_classes represents the total number of unique classes in the target dataset; n_samples j This indicates the total number of rows in the corresponding class.

[0097] The specific implementation idea is as follows: Classify the images in the entire dataset, and then assign a smaller weight to each class as the number of classes. Then, sum the class weights of each image using the total class weights of the entire dataset. In other words, the sampling order is determined by the sum of the class weights of each image. In the code implementation, the images are sorted from largest to smallest.

[0098] When the number of labels in all classes of training images is not equal, it is necessary to change the class weights, thereby changing the image weights. Then, data is re-acquired based on the image weights, which is especially important in cases of imbalanced image classes. When training with YOLOv5, an imbalance in the number of labels for each class is inevitable. To mitigate the impact of class imbalance during training, YOLOv5 introduces the setting of class weights and image weights.

[0099] Furthermore, smaller weights result in smaller penalties and smaller updates to the model coefficients, while larger weights result in larger penalties and larger updates to the model coefficients. Therefore, this invention uses the class weighting method to mitigate the impact of sample imbalance, ultimately obtaining a more accurate output.

[0100] 3. Design a small target detection layer:

[0101] In this embodiment of the invention, the paint defects to be detected have obvious small target attributes. In the YOLOv5 framework, in addition to using the FPN structure to fuse features, a PAN structure is also used. PAN, unlike the FPN network, is an inverted pyramid structure that transmits strong localization features from the bottom up. The two are combined to complete the detection of the target. Since the original YOLOv5 model only has three detection layers, when the input image size is 640*640, the detection layer corresponding to P3 is 80*80 pixels, which can only be used to detect targets larger than 8*8 pixels and lacks the ability to detect smaller targets. This embodiment of the invention modifies the detection head by adding several operation layers. After layer 17, the feature map is further upsampled to expand it. Simultaneously, at layer 20, the obtained 160*160 feature map is fused with the feature map from layer 2 of the backbone network to obtain a larger feature map for small target detection. In layer 31, the detection layer, a small target detection layer is added, making a total of four layers [21, 24, 27, 30] used for detection. The small target detection layer greatly improves the accuracy of the algorithm and enhances its ability to detect small targets.

[0102] 4. Add a set of smaller anchors.

[0103] In the YOLO series of algorithms, specific anchor boxes with varying widths and heights are required for different datasets. During network training, the model outputs corresponding predicted boxes based on the initial anchor boxes, calculates the difference between these predicted boxes and the ground truth boxes, and performs a reverse update operation to update the parameters of the entire network. Therefore, setting the initial anchor boxes is a crucial step. The original YOLOv5 model has only three detection layers, corresponding to three sets of initial anchor values. Anchors can be viewed as pre-defined reference boxes of different sizes and aspect ratios on the image. YOLOv5's anchor mechanism identifies negative examples as follows: except for positive examples, anchors whose IOU with all ground truths is less than a threshold (0.5) are considered negative examples. If the anchor size is much larger than the target size, its positive samples are very likely to be negative. To accommodate the modification of the small target detection layer, a set of anchors is added in this embodiment. This set of anchors is slightly smaller than the original YOLOv5 anchors, with parameters [5, 6, 8, 14, 15, 11].

[0104] In step five above, following step one, cross-validation is used to divide the training dataset into K non-overlapping subsets. In step five, K training and validation iterations are performed. Each time, one subset is used to validate the model, and the other K-1 subsets are used to train the model. Finally, the average of the training and validation errors from these K iterations is calculated. This embodiment of the invention uses this method to divide the dataset into a detection dataset and a segmentation dataset, and divides each dataset into training and validation sets. Furthermore, to accommodate changes in the YOLOv5 model, a corresponding training strategy is designed: alternating training of the detection and segmentation branches. In a batch, forward propagation of the detection and segmentation branches is performed separately. During backpropagation, the weights of the detection and segmentation network branches are updated alternately. This training method makes the model more likely to converge. The trained car paint defect detection and segmentation model, after convergence, can simultaneously detect and segment images containing car paint defects and output the results. See details... Figure 5 As shown.

[0105] This invention discloses an integrated method for vehicle paint defect detection and segmentation based on the YOLOv5 framework. The method obtains both the detected and segmented images of the target vehicle paint image using a vehicle paint defect detection and segmentation model. The model employs the YOLOv5 framework and includes segmentation and detection branches. During the detection and segmentation of the vehicle paint, an FPN network is used to extract and fuse features at different levels, improving both detection and segmentation efficiency and accuracy. This allows for simultaneous vehicle paint defect detection and segmentation, better meeting the needs of practical applications. Segmentation generates a mask image of the defect target; detection reveals the type and location of the defect, while segmentation determines its shape and size.

[0106] The paint defect detection and segmentation model utilizes an FPN network during training. In the bottom-up process of the FPN network, deeper layers focus on semantic information, while shallower layers focus on detailed information. Then, in the top-down process, the detection branch uses information from deeper layers to complete the detection task, addressing the questions of "what is the paint defect?" and "where is the paint defect?". Upsampling further utilizes the detailed information from the bottom-up process to obtain the size and shape of the paint defect. If a direct segmentation network (such as Unet) is used, its structure will be limited by the dataset size, resulting in insufficient feature extraction and a high false negative rate. Therefore, compared to direct segmentation algorithms, this invention uses detection as an aid to segmentation, achieving better results. Furthermore, the method of further integrating detailed information in the detection branch combines semantic and detailed information, ultimately yielding better results.

[0107] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A car paint defect detection and segmentation integrated method based on a YOLOv5 framework, characterized in that, include: S1. Obtain the target vehicle paint image; S2. Input the target paint image into the trained paint defect detection and segmentation model, and output the paint defect detection image and segmentation image corresponding to the target paint image; the paint defect detection and segmentation model adopts the YOLOv5 framework and includes a segmentation branch and a detection branch; In step S2, the training process of the vehicle paint defect detection and segmentation model is as follows: Step 1: Obtain a large number of images of car paint defects with detection and segmentation annotations as a training dataset; and preprocess the training dataset. Step 2: Perform data augmentation on the preprocessed training dataset using the copy-paste method constrained by the k-means clustering method; Step 3: Add a split header to the YOLOv5 framework to obtain split branches; Step 4: Redesign the detection head of the YOLOv5 framework to obtain the detection branches; Step 5: Based on the training dataset after data augmentation, train the segmentation branch and the detection branch in turn until convergence to obtain the trained paint defect detection and segmentation model. Step two specifically includes: using the k-means clustering method to divide the images in the preprocessed training dataset into bright and dark regions; and using the copy-paste method in the bright and dark regions respectively to perform data augmentation processing on the preprocessed training dataset. In step five, the segmentation branch and the detection branch are trained sequentially in turn, specifically including: performing forward propagation of the detection branch and the segmentation branch respectively; and updating the weights in the detection branch and the segmentation branch in turn during the back propagation process.

2. The YOLOv5 framework-based vehicle paint defect detection and segmentation integrated method according to claim 1, characterized in that, In step one, the preprocessing includes: dividing the training dataset using cross-validation to obtain K non-overlapping subsets.

3. The YOLOv5 framework-based vehicle paint defect detection and segmentation integrated method of claim 1, wherein, Step three specifically includes: The YOLOv5 framework uses an SPPF+FPN structure at the neck; at the head of the YOLOv5 framework, the lower layer of the FPN network is fed into the segmentation head to obtain the segmentation branches. Features are extracted from the backbone network of the YOLOv5 framework; The features are fused using an SPPF network to obtain features at different semantic scales; The features at different semantic scales are fed into the FPN network in a top-down upsampling process. The FPN network then splices and fuses the features at different semantic scales. The bottom layer of the FPN network is fed into the segmentation branch, and the segmentation result is obtained after three upsampling operations.

4. The YOLOv5 framework-based vehicle paint defect detection and segmentation integrated method of claim 1, wherein, Step four specifically includes: Add a decoupling header to the YOLOv5 framework; Introducing class weights into the YOLOv5 framework; Add a small-eye detection layer to the detection layer of the YOLOv5 framework; Add a new set of anchor values ​​to the detection layer of the YOLOv5 framework.

5. The integrated method for vehicle paint defect detection and segmentation based on the YOLOv5 framework as described in claim 4, characterized in that, The new anchor value is less than the original anchor value in the YOLOv5 framework.

6. The integrated method for vehicle paint defect detection and segmentation based on the YOLOv5 framework as described in claim 2, characterized in that, In step five, the segmentation branch and the detection branch are trained and validated K times; each time, a subset of the dataset is used to validate the model, and another K-1 subsets of the dataset are used to train the model; finally, the average of the training error and validation error of these K times is calculated.

Images