Door surface scratch detection method based on two-stage deep neural network

Abstract

The application discloses a kind of based on two-stage deep neural network's car door surface scratch detection method, comprising: the surface image of high-resolution car door is collected, and car door background area and car door surface scratch existing car door surface scratch are regarded as a whole, and the target rectangular frame of car door surface area where car door surface scratch is located is labeled;Training set and test set are constructed, and data enhancement is carried out;Two-stage deep neural network is trained using training set, and test set is sent into two-stage deep neural network to output semantic segmentation feature map and classification result;According to semantic segmentation feature map, algorithm is designed to calculate the coordinates of target rectangular frame where car door surface scratch is located in semantic segmentation feature map, to realize the positioning of car door surface scratch.The present application can be applied to the scene that positive and negative samples are extremely imbalanced in automobile production, and high detection accuracy can be maintained under the premise of ensuring rapid detection.

Description

Technical Field

[0001] This invention relates to the technical field of computer vision and automatic defect recognition of scratches on the surface of automobile doors, and in particular to a method for detecting scratches on the surface of automobile doors based on a two-stage deep neural network. Background Technology

[0002] With continuous economic development, users' demands for automobile quality are constantly increasing. Besides meeting daily commuting needs, cars also require a good appearance, and good door surface quality is a prerequisite for this. During the automobile production process, scratches on the door surface are often unavoidable. Because these scratches not only affect the car's aesthetics but can also negatively impact its performance, manufacturers attach great importance to the detection of door surface scratches. This allows them to promptly identify production problems, adjust production strategies, and upgrade production equipment to better meet user needs. Therefore, the detection of door surface scratches is a crucial part of automobile production quality control.

[0003] In the past, manual inspection was the mainstream method for detecting scratches on the surface of car doors. However, this method is not only inefficient, but also difficult for humans to maintain high accuracy and speed when faced with repetitive and heavy inspection work. Machine vision methods based on deep learning can overcome the drawbacks of manual inspection to a large extent.

[0004] With the continuous release of GPUs with stronger computing power, deep learning-based machine vision methods have developed rapidly. More and more researchers are applying these methods to product surface defect detection, such as using object detection methods like YOLO, semantic segmentation methods like FCN, and instance segmentation methods like Mask-RCNN. However, in detecting scratches on car door surfaces, only a small portion of the training samples are positive samples with scratches. Since current deep learning-based machine vision methods often require a large number of training samples, especially positive samples, this significantly limits their application in car door surface scratch detection. Furthermore, the car door surface images acquired in practice are often large-scale, high-resolution images. When these are fed into a deep neural network for training, scaling the image will result in the loss of crucial details due to pixel interpolation. Without scaling, the demands on the device's video memory are very high, significantly increasing the cost of practical applications.

[0005] Based on the above analysis, firstly, although numerous deep learning-based machine vision methods are currently applied to product surface defect detection, their accuracy is often low when applied to car door surface scratch detection due to the inability to obtain sufficient positive samples with scratches, failing to meet the actual needs of automotive production quality control. Secondly, how to utilize large-scale, high-resolution car door surface images for training and improve the detection speed of car door surface scratches remains an unsolved problem in car door surface scratch detection. Therefore, new deep learning-based machine vision methods for car door surface scratch detection are urgently needed to solve these problems. Summary of the Invention

[0006] The purpose of this invention is to overcome the problems of current deep learning-based machine vision methods for detecting scratches on car door surfaces, which struggle to achieve high detection accuracy and speed when training with sufficient positive samples containing scratches, and are also difficult to train using large-scale, high-resolution car door surface images. This invention provides a car door surface scratch detection method based on a two-stage deep neural network. The two-stage deep neural network used in this method can be trained with a small number of positive samples containing scratches, while achieving high detection accuracy and speed. Furthermore, this method reduces the difficulty of training with large-scale, high-resolution car door surface images, saves application costs, and improves practicality.

[0007] To achieve the above objectives, the technical solution provided by this invention is as follows: a method for detecting scratches on the surface of a car door based on a two-stage deep neural network. This two-stage deep neural network consists of a first-stage semantic segmentation network and a second-stage decision network. The first-stage semantic segmentation network follows an encoder-decoder structure, using the encoder to extract low-scale features and high-scale semantic information of the image, and the decoder to restore the semantic segmentation feature map to the scale of the input image and output it. The second-stage decision network is a classifier that incorporates convolutional layers, max-pooling layers, global pooling layers, and fully connected layers. It uses the convolutional layers, max-pooling layers, and global pooling layers to extract features at different scales from the semantic segmentation feature map output by the first-stage semantic segmentation network, and the fully connected layer to classify the scratches on the car door surface and the background, outputting the final classification result.

[0008] The method for detecting scratches on the surface of a car door includes the following steps:

[0009] 1) Acquire high-resolution images of the car door surface, treat the background area of ​​the car door with scratches and the scratches as a whole, and use the annotation tool to annotate the target rectangle of the car door surface area where the scratches are located.

[0010] 2) Construct training and testing sets using labeled images, and perform data augmentation on the training and testing sets to enrich the scale diversity of scratches on the car door surface in the training and testing sets;

[0011] 3) The test set is fed into the pre-trained two-stage deep neural network for detection. The first-stage semantic segmentation network extracts features from the car door surface images in the test set and outputs semantic segmentation feature maps for locating scratches on the car door surface. The second-stage decision network uses the semantic segmentation feature maps output by the first-stage semantic segmentation network to classify the scratches and background on the car door surface and outputs the final classification result.

[0012] 4) Using the semantic segmentation feature map output by the semantic segmentation network in the first stage, an algorithm is designed to calculate the coordinates of the target rectangle containing the scratches on the car door surface in the semantic segmentation feature map, thereby locating the scratches on the car door surface. In this algorithm, there are two cases: single target and multiple targets. For the case of a single target, the coordinates of the first non-zero pixel and the last non-zero pixel in the semantic segmentation feature map are directly calculated as the coordinates of the four vertices of the target rectangle, thereby locating the scratches on the car door surface. For the case of multiple targets, the semantic segmentation feature map is first subjected to median filtering, and then the result of median filtering is subjected to dilation. The OTSU method is used to perform threshold segmentation on the image output after dilation and to find the connected components in the threshold segmentation result. The coordinates of the four vertices of the multiple target rectangles are calculated based on the center coordinates and length and width of the connected components, thereby locating the scratches on the car door surface.

[0013] Furthermore, in step 1), a high-resolution image of the car door surface is acquired using a striped light source and an industrial camera, and the target rectangle of the scratches on the car door surface and its surrounding background is marked using the labelme annotation tool.

[0014] Furthermore, in step 2), data augmentation employs random cropping. When the image is randomly cropped, the marked scratches on the car door surface will change shape, thereby enriching the number of scratches on the car door surface at different angles and scales in the image and improving the network's robustness to different shooting scenarios.

[0015] Furthermore, in step 3), in order to train and achieve high detection accuracy and speed with a small number of positive samples with scratches on the car door surface, a two-stage deep neural network was built. The first-stage semantic segmentation network is responsible for extracting features from the image and outputting semantic segmentation feature maps for locating scratches on the car door surface. The second-stage decision network is responsible for classifying scratches on the car door surface and the background and outputting the final classification result.

[0016] Furthermore, in step 3), the first-stage semantic segmentation network of the dual-stage deep neural network extracts features from the input image and outputs a semantic segmentation feature map for locating scratches on the car door surface. The first-stage semantic segmentation network expands the receptive field of the convolutional layers by employing dilated convolutions, enabling it to extract richer semantic features. Simultaneously, the convolutional layers of the first-stage semantic segmentation network use convolutional kernels of different sizes, allowing the semantic segmentation feature map to contain more high-scale semantic information. The first-stage semantic segmentation network follows an encoder-decoder structure, as detailed below:

[0017] The encoder consists of a two-layer architecture, with each layer consisting of repeated convolution and max pooling operations, as detailed below:

[0018] The first layer architecture consists of two 5×5 convolution operations and one 2×2 max pooling operation to extract features from the input image, as shown in the following formula:

[0019] I1=I0*K5*K5 (1)

[0020] I2=p2(I1) (2)

[0021] In the formula, I0 represents the input image, I1 represents the feature map obtained by repeating the 5×5 convolution operation on I0 twice, I2 represents the feature map obtained by performing a 2×2 max pooling operation on I1, which is also the feature map output by the first architecture, K5 represents the 5×5 convolution operation, and p2 represents the 2×2 max pooling operation.

[0022] The second layer architecture consists of three 5×5 convolution operations and one 2×2 max pooling operation, used to extract features at a scale of half that of the input image. The process is shown in the following formula:

[0023] I3=I2*K5*K5*K5 (3)

[0024] I4=p2(I3) (4)

[0025] In the formula, I3 represents the feature map obtained by repeating the 5×5 convolution operation on I2 three times, and I4 represents the feature map obtained by performing the 2×2 max pooling operation on I3, which is also the feature map output by the encoder.

[0026] The decoder consists of a two-layer architecture, with each layer consisting of repeated convolutional and bilinear interpolation operations, as detailed below:

[0027] The first layer architecture consists of four 5×5 convolution operations and one bilinear interpolation operation, used to extract features from the encoder output feature map and to restore the feature map scale to half the scale of the input image. The process is shown in the following formula:

[0028] I5=I4*K5*K5*K5*K5 (5)

[0029] I6=f linear (I5) (6)

[0030] In the formula, I5 represents the feature map obtained by repeating the 5×5 convolution operation on I4 four times, I6 represents the feature map obtained by performing bilinear interpolation on I5, which is also the feature map output by the first architecture, f linear This represents the bilinear interpolation operation;

[0031] The second layer consists of one 15×15 convolution operation, one 1×1 convolution operation, and one bilinear interpolation operation. It is used to extract features from the output feature map of the decoder's first architecture and to restore the feature map to the same scale as the input image. The process is shown in the following formula:

[0032] I7 = I6 * K 15 *K1 (7)

[0033] I8=f linear (I7) (8)

[0034] In the formula, I7 represents the feature map obtained by performing one 15×15 convolution operation and one 1×1 convolution operation on I6, I8 represents the semantic segmentation feature map obtained by performing bilinear interpolation on I7, which is also the semantic segmentation feature map output by the first-stage semantic segmentation network, K 15 K1 represents a 15×15 convolution operation, and K1 represents a 1×1 convolution operation.

[0035] Furthermore, in step 3), the second-stage decision network of the dual-stage deep neural network comprises a two-layer architecture, as detailed below:

[0036] The first layer consists of three repeated 5×5 convolution operations and 2×2 max pooling operations, used to extract features at different scales from the semantic segmentation feature map output by the first-stage semantic segmentation network. The process is shown in the following formula:

[0037] I1 * =p2(I8*K5) (9)

[0038] I2 * =p2(I1) * *K5) (10)

[0039] I3 * =p2(I2) * *K5) (11)

[0040] In the formula, I1 *I2 represents the feature map obtained by performing one 5×5 convolution operation and one 2×2 max pooling operation on I8. * Indicates I1 * The feature map obtained after one 5×5 convolution operation and one 2×2 max pooling operation, I3 * Indicates I2 * The feature map obtained after one 5×5 convolution operation and one 2×2 max pooling operation is also the feature map output by the first architecture.

[0041] The second-layer architecture consists of one parallel 32×32 global max pooling operation and one 32×32 global average pooling operation, one parallel 1×1 global max pooling operation and one 1×1 global average pooling operation, one concatenation operation along the channel dimension, and one classification operation. The process is shown in the following formula:

[0042] Y1 = p max32 (I3 * )+p avg32 (I3 * (12)

[0043] Y2 = p max1 (I3 * )+p avg1 (I3 * (13)

[0044] Y = f * (Y1,Y2) (14)

[0045] P = FC(Y) (15)

[0046] In the formula, Y1 represents the relationship between I3 and I3. * The feature map obtained after one parallel 32×32 global max pooling operation and one 32×32 global average pooling operation, Y2 represents the feature map of I3. * The feature map obtained after one parallel 1×1 global max pooling operation and one×1 global average pooling operation, Y represents the feature map obtained by concatenating Y1 and Y2 along the channel dimension, and P represents the output classification result. max32 This represents a 32×32 global max pooling operation, p avg32 This represents a 32×32 global average pooling operation, p max1 p represents a 1×1 global max pooling operation. avg1 f represents a 1×1 global average pooling operation. * This indicates a splicing operation along the channel dimension, while FC indicates a classification operation using a fully connected layer.

[0047] Furthermore, in step 3), the two-stage deep neural network is trained end-to-end, and the network loss is calculated using the following formula:

[0048] L fused =αL seg +βL dec (16)

[0049] In the formula, L fused L represents the fusion loss function used in a two-stage deep neural network. seg Let L represent the loss function of the first-stage semantic segmentation network. dec Let L represent the loss function of the second-stage decision network, where α and β represent the weights of the loss function at each stage. seg and L dec Both are cross-entropy loss functions, and their calculation formulas are as follows:

[0050] L cross =-[f(i)·log2p i +(1-f(i))·log2(1-p i (17)

[0051] In the formula, L cross p represents the cross-entropy loss function. i Let f(i) represent the probability that the i-th sample is predicted as a positive sample, and let f(i) represent the sign function, which is calculated as follows:

[0052]

[0053] In formula (16), α and β represent the weights of the loss function at each stage, and they follow the relationship shown in the following formula:

[0054] β=δ(1-α) (19)

[0055] In the formula, δ is a preset hyperparameter. To ensure that the network in both stages is fully trained during end-to-end training, α is set using the following formula to balance the loss of the two stages:

[0056]

[0057] In the formula, E cur E represents the number of training cycles that have been completed so far. totalThis represents the total number of training epochs. To ensure that each stage of the network follows an end-to-end learning strategy, the backpropagation of the second-stage decision network is artificially blocked, thus guaranteeing that the gradient descent processes of the first-stage semantic segmentation network and the second-stage decision network are independent of each other. At the same time, to mitigate the impact of imbalanced positive and negative samples on network training, a sample random sampling method based on usage frequency is used to sample samples in each training iteration during the network training process. This avoids the overuse or underuse of preset samples.

[0058] Furthermore, in step 4), when the semantic segmentation feature map contains only one target, all pixels except the area where the target is located are 0. Therefore, it is only necessary to find the coordinates of non-zero pixels and further filter out the coordinates of the edges as the coordinates of the four vertices of the target rectangle, thereby realizing the localization of the scratches on the car door surface.

[0059] Furthermore, in step 4), when the semantic segmentation feature map contains multiple targets, a 3×3 sliding window is used to perform median filtering on the semantic segmentation feature map.

[0060] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0061] 1. This invention utilizes a first-stage semantic segmentation network to extract features from a car door surface image and output a semantic segmentation feature map for scratch localization. A second-stage decision network is used to classify the scratches and background on the car door surface and output the final classification result. This can transform the training samples from the image level to the pixel level, overcoming the difficulty of training on a small number of positive sample datasets. Furthermore, since the first-stage semantic segmentation network has a large receptive field, it can obtain a semantic segmentation feature map with rich semantics, thereby improving the accuracy of judging the quality of the car door surface and the accuracy of locating scratches on the car door surface.

[0062] 2. This invention improves the detection accuracy of scratches on car door surfaces by performing convolution, downsampling, and global pooling on the semantic segmentation map output by the first-stage semantic segmentation network.

[0063] 3. This invention proposes a method for locating scratches on the surface of a car door based on the features of the semantic segmentation feature map output by the first-stage semantic segmentation network. Compared with some current object detection methods, this method makes full use of the pixel-level features of the semantic segmentation feature map, which can achieve a more accurate localization effect.

[0064] 4. This invention can achieve real-time detection of scratches on the surface of car doors, and the dual-stage deep neural network used does not require a deep backbone network for feature extraction. Therefore, the method of this invention requires less computation and can be applied to embedded devices. Attached Figure Description

[0065] Figure 1 This is a simplified flowchart of the present invention.

[0066] Figure 2 This is a schematic diagram of the network structure of the present invention; in the figure, conv represents convolution operation, pool represents max pooling operation, Bilinear Interpolation represents bilinear interpolation operation, globalmaxpool represents global max pooling operation, globallavgpool represents global average pooling operation, and fully connected layer represents fully connected layer. Detailed Implementation

[0067] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0068] like Figure 1 and Figure 2 As shown, this embodiment provides a method for detecting scratches on the surface of a car door based on a two-stage deep neural network. The two-stage deep neural network consists of a first-stage semantic segmentation network and a second-stage decision network. The first-stage semantic segmentation network follows an encoder-decoder structure, using the encoder to extract low-scale features and high-scale semantic information of the image, and using the decoder to restore the semantic segmentation feature map to the scale of the input image and output it. The second-stage decision network is a classifier that introduces convolutional layers, max pooling layers, global pooling layers, and fully connected layers. It uses the convolutional layers, max pooling layers, and global pooling layers to extract features at different scales from the semantic segmentation feature map output by the first-stage semantic segmentation network, and uses the fully connected layers to classify the scratches on the car door surface and the background and output the final classification result.

[0069] The specific implementation of this method for detecting scratches on the surface of car doors includes the following steps:

[0070] 1) Large-scale, high-resolution images of car door surfaces were acquired using a striped light source and an industrial camera. Images with better image quality were selected to create a dataset. The LabelMe annotation tool was used to annotate the selected car door surface images. During annotation, scratches and their locations were marked with rectangles to obtain .json format label files. The obtained label files were then converted into mask labels for training the first-stage semantic segmentation network.

[0071] 2) Randomly crop the labeled images. Random cropping can reduce the difficulty of training deep networks on large-scale, high-resolution car door surface images. On the other hand, it can simulate different stripe light source brightness and different shooting angles, thereby improving the generalization performance of the network in different automobile production scenarios.

[0072] 3) A two-stage deep neural network is trained using a randomly cropped dataset. The first-stage semantic segmentation network is trained using mask labels from the dataset, while the second-stage decision network is trained using the semantic segmentation feature maps output by the first-stage semantic segmentation network. Using the first-stage semantic segmentation network instead of a deep backbone network for feature extraction effectively reduces the number of parameters and computational load of the two-stage deep neural network, lowering the computational requirements of the hardware and making this method more suitable for application in actual automotive production testing.

[0073] The first-stage semantic segmentation network is used to extract features from the high-resolution car door surface image and output a semantic segmentation feature map for scratch localization. It follows an encoder-decoder structure, as detailed below:

[0074] The encoder consists of a two-layer architecture, with each layer consisting of repeated convolution and max pooling operations, as detailed below:

[0075] The first layer architecture consists of two 5×5 convolution operations and one 2×2 max pooling operation to extract features from the input image, as shown in the following formula:

[0076] I1=I0*K5*K5 (1)

[0077] I2=p2(I1) (2)

[0078] In the formula, I0 represents the input image, I1 represents the feature map obtained by repeating the 5×5 convolution operation on I0 twice, I2 represents the feature map obtained by performing a 2×2 max pooling operation on I1, which is also the feature map output by the first architecture, K5 represents the 5×5 convolution operation, and p2 represents the 2×2 max pooling operation.

[0079] The second layer architecture consists of three 5×5 convolution operations and one 2×2 max pooling operation, used to extract features at a scale of half that of the input image. The process is shown in the following formula:

[0080] I3=I2*K5*K5*K5 (3)

[0081] I4=p2(I3) (4)

[0082] In the formula, I3 represents the feature map obtained by repeating the 5×5 convolution operation on I2 three times, and I4 represents the feature map obtained by performing the 2×2 max pooling operation on I3, which is also the feature map output by the encoder.

[0083] The decoder consists of a two-layer architecture, with each layer consisting of repeated convolutional and bilinear interpolation operations, as detailed below:

[0084] The first layer architecture consists of four 5×5 convolution operations and one bilinear interpolation operation, used to extract features from the encoder output feature map and to restore the feature map scale to half the scale of the input image. The process is shown in the following formula:

[0085] I5=I4*K5*K5*K5*K5 (5)

[0086] I6=f linear (I5) (6)

[0087] In the formula, I5 represents the feature map obtained by repeating the 5×5 convolution operation on I4 four times, I6 represents the feature map obtained by performing bilinear interpolation on I5, which is also the feature map output by the first architecture, f linear This represents the bilinear interpolation operation;

[0088] The second layer consists of one 15×15 convolution operation, one 1×1 convolution operation, and one bilinear interpolation operation. It is used to extract features from the output feature map of the decoder's first architecture and to restore the feature map to the same scale as the input image. The process is shown in the following formula:

[0089] I7 = I6 * K 15 *K1 (7)

[0090] I8=f linear (I7) (8)

[0091] In the formula, I7 represents the feature map obtained by performing one 15×15 convolution operation and one 1×1 convolution operation on I6, I8 represents the semantic segmentation feature map obtained by performing bilinear interpolation on I7, which is also the semantic segmentation feature map output by the first-stage semantic segmentation network, K 15 K1 represents a 15×15 convolution operation, and K1 represents a 1×1 convolution operation.

[0092] Unlike many classic semantic segmentation networks, the first-stage semantic segmentation network does not discard some neurons because the convolutional layers between and within stages share weights, ensuring sufficient regularization. Therefore, it is not necessary to discard some neurons for regularization. To increase the receptive field for high-resolution images, the first-stage semantic segmentation network uses dilated convolutions in each convolutional layer, thereby improving the detection and localization accuracy of scratches on car door surfaces.

[0093] 4) To assess the quality of the predicted car door surface image, the semantic segmentation feature map output from the first-stage semantic segmentation network is input into the second-stage decision network. The second-stage decision network is a classifier incorporating convolutional layers, max-pooling layers, global pooling layers, and fully connected layers. The second-stage decision network consists of a two-layer architecture, as detailed below:

[0094] The first layer consists of three repeated 5×5 convolution operations and 2×2 max pooling operations, used to extract features at different scales from the semantic segmentation feature map output by the first-stage semantic segmentation network. The process is shown in the following formula:

[0095] I1 * =p2(I8*K5) (9)

[0096] I2 * =p2(I1) * *K5) (10)

[0097] I3 * =p2(I2) * *K5) (11)

[0098] In the formula, I1 * I2 represents the feature map obtained by performing one 5×5 convolution operation and one 2×2 max pooling operation on I8. * Indicates I1 * The feature map obtained after one 5×5 convolution operation and one 2×2 max pooling operation, I3 * Indicates I2 * The feature map obtained after one 5×5 convolution operation and one 2×2 max pooling operation is also the feature map output by the first architecture.

[0099] The second-layer architecture consists of one parallel 32×32 global max pooling operation and one 32×32 global average pooling operation, one parallel 1×1 global max pooling operation and one 1×1 global average pooling operation, one concatenation operation along the channel dimension, and one classification operation. The process is shown in the following formula:

[0100] Y1 = p max32 (I3* )+p avg32 (I3 * (12)

[0101] Y2 = p max1 (I3 * )+p avg1 (I3 * (13)

[0102] Y = f * (Y1,Y2) (14)

[0103] P = FC(Y) (15)

[0104] In the formula, Y1 represents the relationship between I3 and I3. * The feature map obtained after one parallel 32×32 global max pooling operation and one 32×32 global average pooling operation, Y2 represents the feature map of I3. * The feature map obtained after one parallel 1×1 global max pooling operation and one×1 global average pooling operation, Y represents the feature map obtained by concatenating Y1 and Y2 along the channel dimension, and P represents the output classification result. max32 This represents a 32×32 global max pooling operation, p avg32 This represents a 32×32 global average pooling operation, p max1 p represents a 1×1 global max pooling operation. avg1 f represents a 1×1 global average pooling operation. * This indicates a splicing operation along the channel dimension, while FC indicates a classification operation using a fully connected layer.

[0105] The design of the second-stage decision network follows two important principles: First, to improve the capacity for capturing scratches and complex shapes on the car door surface, the second-stage decision network employs multi-layer convolution and downsampling operations, enabling the network to perceive the global shape of the image while capturing local shape information. Second, the second-stage decision network utilizes information from the semantic segmentation feature map output by the first-stage semantic segmentation network, avoiding a large number of unwanted interference features and reducing overfitting of many parameters.

[0106] 5) To achieve end-to-end training of the two-stage deep neural network, the loss L of the first-stage semantic segmentation network is... seg Loss L of the second-stage decision network dec Combined into a single loss function L fused The calculation formula is as follows:

[0107] L fused =αL seg +βL dec (16)

[0108] In the formula, L fused L represents the fusion loss function used in a two-stage deep neural network. seg Let L represent the loss function of the first-stage semantic segmentation network. dec Let L represent the loss function of the second-stage decision network, where α and β represent the weights of the loss function at each stage. seg and L dec Both are cross-entropy loss functions, and their calculation formulas are as follows:

[0109] L cross =-[f(i)·log2p i +(1-f(i))·log2(1-p i (17)

[0110] In the formula, L cross p represents the cross-entropy loss function. i Let f(i) represent the probability that the i-th sample is predicted as a positive sample, and let f(i) represent the sign function, which is calculated as follows:

[0111]

[0112] In formula (16), α and β represent the weights of the loss function at each stage, and they follow the relationship shown in the following formula:

[0113] β=δ(1-α) (19)

[0114] In the formula, δ is a preset hyperparameter. To ensure that the network in both stages is fully trained during end-to-end training, α is set using the following formula to balance the loss of the two stages:

[0115]

[0116] In the formula, E cur E represents the number of training epochs that have been completed so far. total This represents the total number of training epochs. The α and β in the formula do not replace the learning rate's corrective effect on network learning in the original loss function; rather, they supplement the learning rate. Introducing α and β allows the network's learning process to be controlled at different scales. More importantly, since most images of the car door surface are normal, scratch-free images, L... dec The resulting classification loss is often compared to L seg The calculated segmentation loss is relatively small, and after introducing δ, the percentage L can be adjusted by setting a preset δ. seg and L dec L fusedThe proportion of classification loss is adjusted to prevent the second-stage decision network from being insufficiently learned due to the neglect of classification loss.

[0117] 6) To further overcome the difficulty of insufficient learning caused by the imbalance of positive and negative samples in the dataset, positive and negative samples are artificially fed into the network alternately in each training iteration during the network training process. In the sampling process of positive and negative samples, a sample random sampling method based on usage frequency is adopted, that is, each sample is sampled with a probability inversely proportional to the usage frequency of the sample. This effectively reduces the overuse or underuse of specific samples during the training process, thereby further improving the detection accuracy of the two-stage deep neural network in the imbalanced positive and negative sample dataset.

[0118] 7) Scratch localization is performed using the semantic segmentation feature map output by the first-stage semantic segmentation network in a two-stage deep neural network. Based on the number of scratches in the car door surface image, the cases can be categorized into single-target and multiple-target scenarios. For a single-target scenario, all pixels except the target area are 0. Therefore, it is only necessary to find the coordinates of non-zero pixels and further filter out the coordinates of the outermost edge as the coordinates of the four vertices of the target rectangle, thus achieving the localization of the scratches on the car door surface. For the multiple-target scenario, a 3×3 sliding window is first used to perform median filtering on the semantic segmentation feature map. Then, dilation is performed on the median-filtered result. Next, the OTSU method is used to perform threshold segmentation on the dilated image. Finally, connected components in the threshold segmentation result are found. Based on the center coordinates and width and height of the connected components, the coordinates of the four vertices of the multiple target rectangles are calculated, thus achieving the localization of the scratches on the car door surface.

[0119] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for detecting scratches on the surface of a car door based on a two-stage deep neural network, characterized in that, In order to train with a small number of positive samples with scratches on the car door surface, a two-stage deep neural network was built, which consists of a first-stage semantic segmentation network and a second-stage decision network. The first stage of the semantic segmentation network follows an encoder-decoder structure. The encoder extracts low-scale features and high-scale semantic information of the image, and the decoder restores the semantic segmentation feature map to the scale of the input image and outputs it. The second-stage decision network is a classifier that incorporates convolutional layers, max pooling layers, global pooling layers, and fully connected layers. It uses convolutional layers, max pooling layers, and global pooling layers to extract features at different scales from the semantic segmentation feature map output by the first-stage semantic segmentation network. It uses fully connected layers to classify scratches on the car door surface and the background and outputs the final classification result. The method for detecting scratches on the surface of a car door includes the following steps: 1) Acquire high-resolution images of the car door surface, treat the background area of ​​the car door with scratches and the scratches as a whole, and use the annotation tool to annotate the target rectangle of the car door surface area where the scratches are located. 2) Construct training and testing sets using labeled images, and perform data augmentation on the training and testing sets to enrich the scale diversity of scratches on the car door surface in the training and testing sets; 3) The test set is fed into the pre-trained two-stage deep neural network for detection. The first-stage semantic segmentation network extracts features from the car door surface images in the test set and outputs semantic segmentation feature maps for locating scratches on the car door surface. The second-stage decision network uses the semantic segmentation feature maps output by the first-stage semantic segmentation network to classify the scratches and background on the car door surface and outputs the final classification result. The first-stage semantic segmentation network of the dual-stage deep neural network extracts features from the input image and outputs a semantic segmentation feature map for locating scratches on the car door surface. The first-stage semantic segmentation network expands the receptive field of the convolutional layer by using dilated convolution, enabling it to extract richer semantic features. At the same time, the convolutional layers of the first-stage semantic segmentation network use convolutional kernels of different sizes, so that the semantic segmentation feature map can contain more high-scale semantic information. The first-stage semantic segmentation network follows an encoder-decoder structure. 4) Using the semantic segmentation feature map output by the semantic segmentation network in the first stage, an algorithm is designed to calculate the coordinates of the target rectangle containing the scratches on the car door surface in the semantic segmentation feature map, thereby locating the scratches on the car door surface. In this algorithm, there are two cases: single target and multiple targets. For the case of a single target, the coordinates of the first non-zero pixel and the last non-zero pixel in the semantic segmentation feature map are directly calculated as the coordinates of the four vertices of the target rectangle, thereby locating the scratches on the car door surface. For the case of multiple targets, the semantic segmentation feature map is first subjected to median filtering, and then the result of median filtering is subjected to dilation. The OTSU method is used to perform threshold segmentation on the image output after dilation and to find the connected components in the threshold segmentation result. The coordinates of the four vertices of the multiple target rectangles are calculated based on the center coordinates and length and width of the connected components, thereby locating the scratches on the car door surface.

2. The method for detecting scratches on the surface of a car door based on a two-stage deep neural network according to claim 1, characterized in that, In step 1), a high-resolution image of the car door surface is acquired using a striped light source and an industrial camera, and the target rectangles of the scratches on the car door surface and the surrounding background are marked using the labelme annotation tool.

3. The method for detecting scratches on the surface of a car door based on a two-stage deep neural network according to claim 2, characterized in that, In step 2), data augmentation uses random cropping. When the image is randomly cropped, the shape of the marked scratches on the car door surface will change, thereby enriching the number of scratches on the car door surface at different angles and scales in the image and improving the robustness of the network to different shooting scenarios.

4. The method for detecting scratches on the surface of a car door based on a two-stage deep neural network according to claim 3, characterized in that, In step 3), the specific details of the first-stage semantic segmentation network are as follows: The encoder consists of a two-layer architecture, with each layer consisting of repeated convolution and max pooling operations, as detailed below: The first layer architecture consists of two 5×5 convolution operations and one 2×2 max pooling operation to extract features from the input image, as shown in the following formula: I1=I0*K5*K5 (1) I2=p2(I1) (2) In the formula, I0 represents the input image, I1 represents the feature map obtained by I0 after repeating the 5×5 convolution operation twice, I2 represents the feature map obtained by I1 after performing the 2×2 max pooling operation, which is also the feature map output by the first architecture, K5 represents the 5×5 convolution operation, and p2 represents the 2×2 max pooling operation. The second layer architecture consists of three 5×5 convolution operations and one 2×2 max pooling operation, used to extract features at a scale of half that of the input image. The process is shown in the following formula: I3=I2*K5*K5*K5 (3) I4=p2(I3) (4) In the formula, I3 represents the feature map obtained by I2 after repeating the 5×5 convolution operation 3 times, and I4 represents the feature map obtained by I3 after performing the 2×2 max pooling operation, which is also the feature map output by the encoder. The decoder consists of a two-layer architecture, with each layer consisting of repeated convolutional and bilinear interpolation operations, as detailed below: The first layer architecture consists of four 5×5 convolution operations and one bilinear interpolation operation, used to extract features from the encoder output feature map and to restore the feature map scale to half the scale of the input image. The process is shown in the following formula: I5=I4*K5*K5*K5*K5 (5) I6=f linear (I5) (6) In the formula, I5 represents the feature map obtained by repeating the 5×5 convolution operation 4 times with I4, I6 represents the feature map obtained by bilinear interpolation of I5, which is also the feature map output by the first architecture, f linear This represents the bilinear interpolation operation; The second layer consists of one 15×15 convolution operation, one 1×1 convolution operation, and one bilinear interpolation operation. It is used to extract features from the output feature map of the decoder's first architecture and to restore the feature map to the same scale as the input image. The process is shown in the following formula: I7=I6*K 15 *K1 (7) I8=f linear (I7) (8) In the formula, I7 represents the feature map obtained by I6 after one 15×15 convolution operation and one 1×1 convolution operation, I8 represents the semantic segmentation feature map obtained by bilinear interpolation of I7, which is also the semantic segmentation feature map output by the first-stage semantic segmentation network, K 15 K1 represents a 15×15 convolution operation, and K1 represents a 1×1 convolution operation.

5. The method for detecting scratches on the surface of a car door based on a two-stage deep neural network according to claim 4, characterized in that, In step 3), the second-stage decision network of the two-stage deep neural network consists of a two-layer architecture, as detailed below: The first layer consists of three repeated 5×5 convolution operations and 2×2 max pooling operations, used to extract features at different scales from the semantic segmentation feature map output by the first-stage semantic segmentation network. The process is shown in the following formula: I1 * =p2(I8*K5) (9) I2 * =p2(I1 * *K5) (10) I3 * =p2(I2 * *K5) (11) In the formula, I1 * I8 represents the feature map obtained after one 5×5 convolution operation and one 2×2 max pooling operation, and I2 represents the feature map obtained after that. * Indicates I1 * The feature map obtained after one 5×5 convolution operation and one 2×2 max pooling operation, I3 * Indicates I2 * The feature map obtained after one 5×5 convolution operation and one 2×2 max pooling operation is also the feature map output by the first architecture. The second-layer architecture consists of one parallel 32×32 global max pooling operation and one 32×32 global average pooling operation, one parallel 1×1 global max pooling operation and one 1×1 global average pooling operation, one concatenation operation along the channel dimension, and one classification operation. The process is shown in the following formula: Y1=p max32 (I3 * )+p avg32 (I3 * ) (12) Y2=p max1 (I3 * )+p avg1 (I3 * ) (13) Y=f * (Y1,Y2) (14) P = FC(Y) (15) In the formula, Y1 represents I3 * The feature map obtained after one parallel 32×32 global max pooling operation and one 32×32 global average pooling operation, Y2 represents I3. * The feature map obtained after one parallel 1×1 global max pooling operation and one×1 global average pooling operation, Y represents the feature map obtained by concatenating Y1 and Y2 along the channel dimension, and P represents the output classification result. max32 This represents a 32×32 global max pooling operation, p avg32 This represents a 32×32 global average pooling operation, p max1 p represents a 1×1 global max pooling operation. avg1 f represents a 1×1 global average pooling operation. * This indicates a splicing operation along the channel dimension, while FC indicates a classification operation using a fully connected layer.

6. The method for detecting scratches on the surface of a car door based on a two-stage deep neural network according to claim 5, characterized in that, In step 3), the two-stage deep neural network is trained end-to-end, and the network loss is calculated using the following formula: L fused =αL seg +βL dec (16) In the formula, L fused L represents the fusion loss function used in a two-stage deep neural network. seg Let L represent the loss function of the first-stage semantic segmentation network. dec Let L represent the loss function of the second-stage decision network, where α and β represent the weights of the loss function at each stage. seg and L dec Both are cross-entropy loss functions, and their calculation formulas are as follows: L cross =-[f(i)·log2 p i +(1-f(i))·log2(1-p i )] (17) In the formula, L cross p represents the cross-entropy loss function. i Let f(i) represent the probability that the i-th sample is predicted as a positive sample, and let f(i) represent the sign function, which is calculated as follows: In formula (16), α and β follow the relationship shown in the following equation: β=δ(1-α) (19) In the formula, δ is a preset hyperparameter. To ensure that the network in both stages is fully trained during end-to-end training, α is set using the following formula to balance the loss of the two stages: In the formula, E cur E represents the number of training cycles that have been completed so far. total This represents the total number of training epochs. To ensure that each stage of the network follows an end-to-end learning strategy, the backpropagation of the second-stage decision network is artificially blocked, thus guaranteeing that the gradient descent processes of the first-stage semantic segmentation network and the second-stage decision network are independent of each other. To mitigate the impact of imbalanced positive and negative samples on network training, a sample random sampling method based on usage frequency is used to sample samples in each training iteration during the network training process. This avoids the overuse or underuse of preset samples.

7. The method for detecting scratches on the surface of a car door based on a two-stage deep neural network according to claim 6, characterized in that, In step 4), when the semantic segmentation feature map contains multiple targets, a 3×3 sliding window is used to perform median filtering on the semantic segmentation feature map.

Images