Vehicle appearance defect detection method, device and storage medium
The CNN-based vehicle appearance defect detection method addresses inefficiencies in traditional methods by enhancing data volume and accuracy through augmentation and clustering, achieving automated and cost-effective defect detection.
Patent Information
- Application Number
- CN202110823954.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-21
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2041-07-21
AI Technical Summary
Existing defect detection methods in the industry face challenges such as low efficiency, poor accuracy, high cost, and inability to fully automate production due to manual sampling, human uncertainty, and limited reusability of traditional machine vision algorithms, exacerbated by scarce defect detection data sets and the need for different treatments of varying defect situations.
A vehicle appearance defect detection method utilizing a convolutional neural network (CNN) approach, involving data collection, augmentation, clustering, and network training to enhance data volume and accuracy, using an hourglass network, bidirectional multi-scale feature network, and skip-connected convolutional detection branches for robust defect detection.
The method automates defect detection, reducing labor costs and improving efficiency while maintaining high accuracy, enabling full-scale detection without sampling and minimizing human error.
Smart Images
Figure CN113570565B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing and deep learning technology, and more specifically, to a vehicle appearance defect detection method, device and computer-readable storage medium. Background Art
[0002] Defect detection, as the name implies, is a technology for locating and classifying defects in objects. It is widely used in all corners of the industry. At present, most defect detection methods in the industry are manual sampling followed by observation and measurement by professional employees. Such defect detection methods require huge manual training and follow-up costs, and human uncertainty leads to huge fluctuations in detection efficiency and accuracy. In addition, sampling cannot include all products, and sampling will interrupt the automation process, resulting in the inability to fully automate production. This series of reasons has made defect detection a pain point in the industry.
[0003] Due to the variety of defects, it is difficult for traditional machine vision algorithms to fully model and migrate defect features, and the reusability is very low. In addition, due to the variety of situations of the same defect, they need to be treated differently, which often wastes a lot of manpower and material resources. In addition, defect detection data sets are very rare, and those involving industrial secrets are often not disclosed. Summary of the invention
[0004] In view of this, the present disclosure proposes a vehicle appearance defect detection method, device and computer-readable storage medium to solve the problems of low efficiency, poor accuracy, high cost and slow speed of existing methods.
[0005] According to a first aspect of the present disclosure, a vehicle appearance defect detection method is proposed, comprising:
[0006] Collect vehicle appearance defect images and normal vehicle appearance images, annotate the collected images, and build a vehicle appearance dataset;
[0007] Performing data preprocessing and data set division on the vehicle appearance data set, using data augmentation and clustering methods to increase the data volume of the vehicle appearance data set, and summarizing data characteristics;
[0008] Build a vehicle appearance defect detection network and train it using preprocessed parameters and data;
[0009] Select the detection method according to the needs, import the image to be detected into the trained vehicle appearance defect detection network, output the defect area and defect type, and realize vehicle appearance defect detection based on convolutional neural network.
[0010] Optionally, the construction of the vehicle appearance dataset includes:
[0011] Collect appearance defect images of the vehicle to be inspected;
[0012] The position of the vehicle appearance defect is marked on the vehicle appearance defect image to obtain a defect marking box, which includes the defect location and the defect, and the defect marking box is stored in a vehicle appearance dataset. The vehicle appearance dataset contains the vehicle appearance defect image and the normal vehicle appearance image, the coordinates of the center point in the defect marking box, the length and width of the marking box, and the marking box category.
[0013] Optionally, the data augmentation and clustering method comprises:
[0014] The vehicle appearance defect images and normal vehicle appearance images in the vehicle appearance dataset are randomly rotated, scaled, brightness changed and blurred to obtain the vehicle appearance defect images and normal vehicle appearance images after augmentation transformation, and the vehicle appearance defect images and normal vehicle appearance images after augmentation transformation are added to the vehicle appearance dataset;
[0015] Processing the data set after the augmentation transformation, cropping the defect annotation frames whose area is less than 1% of the entire image area according to the defect position, and randomly pasting the cropped defects to other background areas without defects in the same image for multiple times to obtain multiple defect annotation frames, and storing the center point coordinates, length and width of the annotation frames, and the annotation frame categories of the multiple defect annotation frames in the vehicle appearance data set respectively;
[0016] The ratios of the length and width of multiple defect annotation boxes in the vehicle appearance dataset to the length and width of the corresponding images are respectively counted; the multiple length-width ratios are clustered using the K-Means method to obtain nine pre-selected anchor boxes, and the nine pre-selected anchor boxes are sorted in sequence according to size; the three smallest boxes among the nine pre-selected anchor boxes after sorting are used as anchor boxes for feature maps downsampled 8 times, three medium-sized boxes are used as anchor boxes for feature maps downsampled 16 times, and the three largest boxes are used as anchor boxes for feature maps downsampled 32 times, so as to obtain the characteristics of the data in the vehicle appearance dataset.
[0017] Optionally, the network construction and training includes the following steps:
[0018] Construct a vehicle appearance defect detection network, including an hourglass network, a bidirectional multi-scale feature network, and three skip-connected convolutional detection network branches;
[0019] 10% of the data in the vehicle appearance dataset is used as a test set, and 90% is used as a training set;
[0020] The vehicle appearance defect detection network is trained using the training set data in the vehicle appearance dataset.
[0021] Optionally, the training of the vehicle appearance defect detection network comprises the following steps:
[0022] The input data volume of the vehicle appearance defect detection network is set. According to the input data volume, the maximum length and maximum width of each batch of images input into the vehicle appearance defect detection network during the training process are determined, and black pixels are filled in all directions of each input image to make the length and width of all input images in the same batch consistent;
[0023] The input image is input into the hourglass network, the hourglass network outputs an image of a region of interest corresponding to the input image, the image of the region of interest is regarded as a heat map of the input image, a threshold of the heat map is set, and the input image is cropped according to the set threshold to obtain an image that may have defects;
[0024] Input the image that may have defects into a bidirectional multi-scale feature network, and the bidirectional multi-scale feature network outputs a multi-scale network downsampled 8 times feature map, a downsampled 16 times feature map, and a downsampled 32 times feature map of the input image in the step;
[0025] The downsampled 8-fold feature map, the downsampled 16-fold feature map, and the downsampled 32-fold feature map of the multi-scale network are respectively input into three skip connection convolutional detection network branches, and the three skip connection convolutional detection network branches perform defect detection on the downsampled 8-fold feature map, the downsampled 16-fold feature map, and the downsampled 32-fold feature map according to the characteristics of the data in the vehicle appearance data set to obtain an image prediction box that may have defects, and the image prediction box that may have defects is a classification score that includes the coordinates of the center point of the prediction box and the length and width of the prediction box;
[0026] A non-maximum suppression algorithm is adopted, and a non-maximum suppression algorithm threshold is set. The classification scores of the image prediction frames that may have defects are screened, and the image prediction frames whose classification scores are greater than the threshold are selected to obtain the final detected defect detection frames. A loss function is used to calculate the loss of the detected defect detection frames and the labeled frames, and the gradients are fed back to the vehicle appearance defect detection network to complete the training of the vehicle appearance defect detection network.
[0027] Optionally, the convolution layer in the feature extraction stage of the bidirectional multi-scale feature network adopts the LeakyRelu activation function, and performs the first feature map downsampling through spatial depth conversion, that is, converting a feature map into four images with the same length and width as the original feature map. Figure 2 One-tenth of the image is used to achieve feature map downsampling;
[0028] The bidirectional multi-scale feature network uses the pyramid pooling method instead of ordinary pooling. The specific operation is: the feature channel is divided into four parts, and then spliced after passing through pooling layers with different pooling rates. The pooling rate is designed according to the actual situation of the network.
[0029] Optionally, the detection output layer activation function of the skip connection convolutional detection network adopts a SWISH activation function;
[0030] The skip connection convolution detection network uses the pyramid pooling method instead of ordinary pooling. The specific operation is to divide the feature channel into four parts, pass through pooling layers with different pooling rates, and then splice them. The pooling rate is designed according to the actual situation of the network.
[0031] Optionally, the detection method comprises:
[0032] (1) When the vehicle appearance defect detection speed is greater than 30 frames per second, the vehicle appearance image to be inspected is scaled and input into the bidirectional multi-scale network in the vehicle appearance defect detection network; when the vehicle appearance defect detection speed is less than or equal to 30 frames per second, the vehicle appearance image to be inspected is input into the hourglass network in the vehicle appearance defect detection network for cropping, and the cropped image is input into the bidirectional multi-scale network;
[0033] (2) Calculate the minimum scaling size required for downsampling the appearance image of the vehicle to be inspected by 32 times, then fill black pixels around the appearance image of the vehicle to be inspected to make the appearance image of the vehicle to be inspected reach the calculated size, and use the appearance image of the vehicle to be inspected as the input of the vehicle appearance detection network to obtain the final detected defect detection frame;
[0034] (3) The non-maximum suppression method is used to select the prediction boxes with a classification score of more than 50% from the final detected defect detection boxes as defect detection boxes, thereby realizing the detection of vehicle appearance defects based on convolutional neural networks.
[0035] According to a second aspect of the present disclosure, a vehicle appearance defect detection device is provided, comprising:
[0036] An image acquisition module is used to acquire vehicle appearance defect images and normal vehicle appearance images, and to annotate the acquired vehicle appearance defect images and normal vehicle appearance images respectively, so as to construct a vehicle appearance data set;
[0037] An image processing module, used for performing data preprocessing and data set division on the vehicle appearance data set, using data augmentation and clustering methods to increase the data volume of the vehicle appearance data set, and summarizing data characteristics;
[0038] The network construction and training module is used to build a vehicle appearance defect detection network and use the preprocessed parameters and data for training;
[0039] The defect detection module is used to select the detection method according to the needs, import the image to be detected into the trained vehicle appearance defect detection network, output the defect area and defect type, and realize vehicle appearance defect detection based on convolutional neural network.
[0040] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program is used to enable the computer to execute the vehicle appearance defect detection method described in any one of claims 1-8.
[0041] According to the embodiments of the present disclosure, the present disclosure can automatically determine whether there are specific defects in the vehicle appearance picture and predict the location and category of the defects, which greatly saves labor costs, improves detection efficiency, and can save a lot of money for enterprises, and has important practical application value.
[0042] The above description is only an overview of one embodiment of the present disclosure. In order to more clearly understand the technical means of the present invention so that it can be implemented in accordance with the contents of the specification, and in order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is a flow chart of a vehicle appearance defect detection method in one embodiment of the present disclosure.
[0044] Figure 2 It is a network structure diagram of a vehicle appearance defect detection method in one embodiment of the present disclosure.
[0045] Figure 3 It is a structural block diagram of a vehicle appearance defect detection device in one embodiment of the present disclosure. DETAILED DESCRIPTION
[0046] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present disclosure to those skilled in the art. All other embodiments obtained by those of ordinary skill in the art without making creative efforts are within the scope of protection of the present disclosure. The present disclosure uses an embodiment of vehicle appearance defect detection based on YOLOv5.
[0047] Figure 1 It is a flowchart showing a vehicle appearance defect detection method according to an embodiment of the present disclosure. The vehicle appearance defect detection method shown in this embodiment can be applicable to user devices, such as mobile phones, tablet computers, etc.
[0048] like Figure 1 As shown, the method may include the following steps:
[0049] In step S1, vehicle appearance defect images and normal vehicle appearance images are collected, and data annotation is performed on the collected images to construct a vehicle appearance data set; the vehicle appearance defect detection data set is divided into two categories according to a preset ratio: a test set and a training set. The ratio of the training set to the test set is 90% and 10%.
[0050] In one embodiment, a camera is used to shoot the appearance defects of the vehicle to be detected, and an appearance defect image of the vehicle to be detected is collected. In one embodiment of the present disclosure, the detected appearance defects of the vehicle include headlight defects, vehicle dashboard defects, appearance scratch defects, parking brake appearance defects, etc.;
[0051] The position of the vehicle appearance defect is marked on the vehicle appearance defect image to obtain a defect marking frame. In one embodiment of the present disclosure, the position of the vehicle appearance defect is manually marked, and the large defect area and the precise defect location are marked to construct a vehicle appearance defect detection data set. The defect marking frame includes the defect location, which refers to the location of the defect on the vehicle, such as the front of the vehicle, the roof, etc., and the precise defect location. The defect marking frame is stored in a vehicle appearance data set, which includes the vehicle appearance defect image and the normal vehicle appearance image, the coordinates of the center point in the defect marking frame, the length and width of the marking frame, and the marking frame category.
[0052] In step S2, data preprocessing and data set division are performed on the vehicle appearance data set, and data augmentation and clustering methods are used to increase the data volume of the vehicle appearance data set, and the data characteristics are summarized;
[0053] In one embodiment of the present disclosure, a method for data augmentation and clustering includes:
[0054] The vehicle appearance defect images and normal vehicle appearance images in the vehicle appearance dataset are randomly rotated, scaled, brightness changed and blurred to obtain the vehicle appearance defect images and normal vehicle appearance images after augmentation transformation, and the vehicle appearance defect images and normal vehicle appearance images after augmentation transformation are added to the vehicle appearance dataset;
[0055] Processing the data set after the augmentation transformation, cropping the defect annotation frames whose area is less than 1% of the entire image area according to the defect position, and randomly pasting the cropped defects to other background areas without defects in the same image for multiple times to obtain multiple defect annotation frames, and storing the center point coordinates, length and width of the annotation frames, and the annotation frame categories of the multiple defect annotation frames in the vehicle appearance data set respectively;
[0056] The ratios of the length and width of multiple defect annotation boxes in the vehicle appearance dataset to the length and width of the image to which they belong are counted respectively; the multiple length and width ratios are clustered using the K-Means method to obtain nine pre-selected anchor boxes, which are sorted in order according to size, and the three smallest boxes among the nine pre-selected anchor boxes after sorting are used as anchor boxes for feature maps that are downsampled 8 times, three medium-sized boxes are used as anchor boxes for feature maps that are downsampled 16 times, and the three largest boxes are used as anchor boxes for feature maps that are downsampled 32 times, to obtain the characteristics of the data in the vehicle appearance dataset. In actual use, the anchor boxes will be multiplied by the length and width of the image and rounded before use.
[0057] In step S3, a vehicle appearance defect detection network is constructed and trained using preprocessed parameters and data. For each batch of input images, its maximum length and width are determined, and then black pixels are evenly filled in all directions of each image to achieve a uniform length and width; the hourglass image is used to output the image of the region of interest, and the image is cropped according to the result heat map to crop out the parts that may have defects.
[0058] By means of the above technical solution, the technical solution disclosed in the present invention has at least the following advantages:
[0059] In the embodiments of the present disclosure, the defect detection algorithm based on the convolutional neural network is based on a one-stage network structure. Data augmentation and clustering methods are added to increase the amount of data and summarize the data characteristics, and the anchor points and scaling sizes are adaptively designed to make the network more robust; adding a secondary cropping network can pay more attention to the area of interest and assist in detection; through the fusion of multi-scale networks and multi-dimensional features, the network ensures the ability to detect small defects while retaining the ability to detect large defects, and the proportion can be artificially adjusted according to the data set. Use a new convolution module to increase the ability to extract features, and use a new pooling module to reduce the information loss of downsampling; improve the loss function, pay more attention to the IOU indicator, and let the loss function better guide network training.
[0060] In one embodiment, a vehicle appearance defect detection network is constructed and trained using preprocessed parameters and data, including:
[0061] Construct a vehicle appearance defect detection network, whose network structure is as follows Figure 2 As shown, it includes an hourglass network, a bidirectional multi-scale feature network, and three skip connection convolutional detection network branches;
[0062] 10% of the data in the vehicle appearance dataset is used as a test set, and 90% is used as a training set;
[0063] The vehicle appearance defect detection network is trained using the training set data in the vehicle appearance dataset.
[0064] In one embodiment, training a vehicle appearance defect detection network includes the following steps:
[0065] According to the memory size of the graphics card on which the network runs, the input data volume of the vehicle appearance defect detection network is set. Based on the input data volume, the maximum length and maximum width of each batch of images input into the vehicle appearance defect detection network during the training process are determined, and black pixels are filled in all directions of each input image to make the length and width of all input images in the same batch consistent; the maximum length and maximum width are determined by different images.
[0066] The input image is input into the hourglass network, the hourglass network outputs an image of a region of interest corresponding to the input image, the image of the region of interest is regarded as a heat map of the input image, a threshold of the heat map is set, and the input image is cropped according to the set threshold to obtain an image that may have defects;
[0067] Input the image that may have defects into a bidirectional multi-scale feature network, and the bidirectional multi-scale feature network outputs a multi-scale network downsampled 8 times feature map, a downsampled 16 times feature map, and a downsampled 32 times feature map of the input image in the step;
[0068] The downsampled 8-fold feature map, the downsampled 16-fold feature map, and the downsampled 32-fold feature map of the multi-scale network are respectively input into three skip connection convolutional detection network branches, and the three skip connection convolutional detection network branches perform defect detection on the downsampled 8-fold feature map, the downsampled 16-fold feature map, and the downsampled 32-fold feature map according to the characteristics of the data in the vehicle appearance data set to obtain an image prediction box that may have defects, and the image prediction box that may have defects is a classification score that includes the coordinates of the center point of the prediction box and the length and width of the prediction box;
[0069] A non-maximum suppression algorithm is adopted, and a non-maximum suppression algorithm threshold is set. The classification scores of the image prediction frames that may have defects are screened, and the image prediction frames whose classification scores are greater than the threshold are selected to obtain the final detected defect detection frames. A loss function is used to calculate the loss of the detected defect detection frames and the labeled frames, and the gradients are fed back to the vehicle appearance defect detection network to complete the training of the vehicle appearance defect detection network.
[0070] In one embodiment, the convolutional layer in the feature extraction stage in the bidirectional multi-scale feature network uses the LeakyRelu activation function, which is expressed as:
[0071]
[0072] Among them, x represents the input feature map, and α is a manually set hyperparameter;
[0073] The bidirectional multi-scale feature network performs the first feature map downsampling through spatial depth conversion, that is, converting a feature map into four maps with the same length and width as the original feature map. Figure 2 One-tenth of the image is used to achieve feature map downsampling;
[0074] The bidirectional multi-scale feature network uses the pyramid pooling method instead of ordinary pooling. The specific operation is to divide the feature channel into four parts, pass through pooling layers with different pooling rates, and then splice them. The pooling rate is designed according to the actual situation of the network.
[0075] In this embodiment, the images in the vehicle appearance data set are randomly rotated, scaled, brightness changed and blurred to obtain the transformed images, and the transformed images are added to the vehicle appearance data set. The annotation boxes in the augmented data set are determined, and the annotation boxes whose area is less than 1% of the entire image area are cropped according to the position of the annotation boxes, and the cropped defective parts are randomly pasted to other background areas of the same image that do not have defects, and the coordinates, length, width and category of the new box are recorded and added to the annotation box data to obtain the vehicle appearance data set after data augmentation.
[0076] In this embodiment, the ratio of the length and width of each annotation box in the vehicle appearance data set to the length and width of the image to which the annotation box belongs is counted to obtain the annotation box ratio data, and the K-Means method is used for clustering to obtain nine pre-selected anchor boxes as training parameters. The nine pre-selected anchor boxes are sorted according to size to obtain the three smallest boxes as the anchor boxes of the feature map downsampled 8 times, the three medium-sized boxes as the anchor boxes of the feature map downsampled 16 times, and the three largest boxes as the anchor boxes of the feature map downsampled 32 times. When actually used, the anchor boxes will be multiplied by the length and width of the image and rounded before use.
[0077] In this embodiment, the defect detection network first performs a downsampling operation on the image, and through the method of spatial conversion channels, the number of channels is quadrupled and the length and width are reduced by two times to obtain a downsampled image.
[0078] The network uses a bidirectional multi-scale network, such as Figure 2 As shown in FIG. 1 , the downsampled features are fused with the upsampled features, and finally the feature maps downsampled by 8 times, 16 times, and 32 times are selected for fusion and detection.
[0079] Pooling is performed through the pyramid pooling layer. Pyramid pooling uses four pooling rates, namely 1 / 5 / 9 / 13, and pooling uses the maximum pooling operation.
[0080] In one embodiment, the detection output layer activation function of the skip connection convolutional detection network adopts the SWISH activation function, which is expressed as:
[0081] swish(x)=x*sigmod(βx)
[0082] Among them, x represents the input feature map, β is a manually set hyperparameter, and the sigmoid formula is:
[0083]
[0084] Among them, x represents the input feature map, and e is the natural logarithm;
[0085] The skip connection convolution detection network uses the pyramid pooling method instead of ordinary pooling. The specific operation is to divide the feature channel into four parts, pass through pooling layers with different pooling rates, and then splice them. The pooling rate is designed according to the actual situation of the network.
[0086] In one embodiment, the loss function used in the vehicle appearance defect detection network training is expressed as:
[0087]
[0088] Where Distance is the distance between the final detected defect detection box and the center point of the corresponding annotation box in the vehicle appearance defect dataset. It represents the diagonal length of the minimum bounding box of the final detected defect detection box and the corresponding annotation box in the vehicle appearance defect dataset. IOU is the intersection-over-union ratio of the final detected defect detection box and the corresponding annotation box in the vehicle appearance defect dataset, and v is an artificially defined parameter to measure the difference between the annotation box and the true box. The calculation formula is:
[0089]
[0090] Where W and H are the length and width of the final defect detection frame, W G , H G is the length and width of the corresponding annotation box in the vehicle appearance defect dataset, and the total loss function is:
[0091] L=L focal +αL classificition +βL IOU +L Hard
[0092] Where L focal is the secondary cropping loss, α and β are artificially defined to adjust the loss ratios, which are set to 1 and 2 respectively in this method. The second loss is the classification loss, using the softmax loss, the third is the positioning loss mentioned above, and the fourth is the difficult sample loss, where L focal The calculation formula is:
[0093]
[0094] Where y represents the labeled value, represents the predicted value, γ is a hyperparameter set manually, and L Hard Represents the loss of difficult samples. The loss of difficult samples means increasing the loss ratio of difficult samples for samples that failed to be predicted. In this method, the loss of difficult samples is doubled.
[0095] In step S4, a detection method is selected according to demand, the image to be detected is imported into the trained vehicle appearance defect detection network, the defect area and defect type are output, and vehicle appearance defect detection based on convolutional neural network is realized.
[0096] In one embodiment, the detection method comprises:
[0097] When the vehicle appearance defect detection speed is greater than 30 frames per second (FPS), the appearance image of the vehicle to be inspected is scaled and input into the bidirectional multi-scale network in the vehicle appearance defect detection network; when the vehicle appearance defect detection speed is less than or equal to 30 frames per second (FPS), that is, when the speed is not required to be greater than 30 FPS, the appearance image of the vehicle to be inspected is input into the hourglass network in the vehicle appearance defect detection network for cropping, and the cropped image is input into the bidirectional multi-scale network;
[0098] Calculate the minimum scaling size required for downsampling the appearance image of the vehicle to be inspected by 32 times, then fill black pixels around the appearance image of the vehicle to be inspected to make the appearance image of the vehicle to be inspected reach the calculated size, and use the appearance image of the vehicle to be inspected as the input of the vehicle appearance detection network to obtain the final detected defect detection frame;
[0099] The non-maximum suppression method selects the prediction box with a classification score of more than 50% from the final detected defect detection box as the defect detection box, realizing the detection of vehicle appearance defects based on convolutional neural network.
[0100] Corresponding to the above-mentioned embodiment of the vehicle appearance defect detection method, the present disclosure proposes an embodiment of a vehicle appearance defect detection device.
[0101] Figure 3 : is a structural block diagram of an embodiment of a vehicle appearance defect detection device proposed in the present disclosure, comprising:
[0102] An image acquisition module is used to acquire vehicle appearance defect images and normal vehicle appearance images, and to annotate the acquired vehicle appearance defect images and normal vehicle appearance images respectively, so as to construct a vehicle appearance data set;
[0103] An image processing module, used for performing data preprocessing and data set division on the vehicle appearance data set, using data augmentation and clustering methods to increase the data volume of the vehicle appearance data set, and summarizing data characteristics;
[0104] The network construction and training module is used to build a vehicle appearance defect detection network and use the preprocessed parameters and data for training;
[0105] The defect detection module is used to select the detection method according to the needs, import the image to be detected into the trained vehicle appearance defect detection network, output the defect area and defect type, and realize vehicle appearance defect detection based on convolutional neural network.
[0106] In this embodiment, the input image size is about 608*608, which will fluctuate with the input, and the output features of three scales are 76*76, 38*38, and 19*19 respectively.
[0107] Select the boxes with high scores on the three scales for non-maximum suppression screening, and finally obtain the defect location and classification that meet the requirements. In this embodiment, the screening score threshold is 0.5, the screening IOU threshold is 0.7, and the upper limit of the screening number is 20.
[0108] The network training disclosed herein uses the Adam optimizer, and the learning rate uses 0.01.
[0109] The prediction method used in this embodiment is to predict one picture at a time, and the speed can reach 50FPS. This method is low-cost and only requires the use of manually labeled data and program running hardware. Vehicle appearance defect pictures can be obtained from the repair picture library, and the present invention can perform full-scale detection without sampling, and the accuracy will not fluctuate with employees.
[0110] The embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is used to cause the computer to execute:
[0111] Collect vehicle appearance defect images and normal vehicle appearance images, annotate the collected images, and build a vehicle appearance dataset;
[0112] Performing data preprocessing and data set division on the vehicle appearance data set, using data augmentation and clustering methods to increase the data volume of the vehicle appearance data set, and summarizing data characteristics;
[0113] Build a vehicle appearance defect detection network and train it using preprocessed parameters and data;
[0114] Select the detection method according to the needs, import the image to be detected into the trained vehicle appearance defect detection network, output the defect area and defect type, and realize vehicle appearance defect detection based on convolutional neural network.
[0115] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0116] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0117] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: receives a request from a first user to configure boot information for the access party software; provides a configuration page for the first user according to the request; obtains the display conditions and display location information set by the first user for the boot information on the configuration page; saves the display conditions and display location information of the boot information in the boot configuration system; and when it is monitored that the operation of the access party software meets the display conditions, controls the access party software to display the boot information according to the display location information.
[0118] Alternatively, the computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: receives a request from a first user to configure boot information for the access party software; provides a configuration page for the first user based on the request; obtains display conditions and display location information set by the first user for the boot information on the configuration page; saves the display conditions and display location information of the boot information in the boot configuration system; and when it is monitored that the operation of the access party software meets the display conditions, controls the access party software to display the boot information based on the display location information.
[0119] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0120] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0121] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
[0122] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0123] A person skilled in the art may understand that all or part of the steps in the above-mentioned embodiment method may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0124] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0125] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A vehicle appearance defect detection method, characterized in that: include: Collect vehicle appearance defect images and normal vehicle appearance images, annotate the collected images, and build a vehicle appearance dataset; Performing data preprocessing and data set division on the vehicle appearance data set, using data augmentation and clustering methods to increase the data volume of the vehicle appearance data set, and summarizing data characteristics; Build a vehicle appearance defect detection network and train it using preprocessed parameters and data; Select a detection method, import the image to be detected into the trained vehicle appearance defect detection network, and output the defect area and defect type; The construction of a vehicle appearance defect detection network, using pre-processed parameters and data for training, includes: Construct a vehicle appearance defect detection network, including an hourglass network, a bidirectional multi-scale feature network, and three skip-connected convolutional detection network branches; 10% of the data in the vehicle appearance dataset is used as a test set, and 90% is used as a training set; Use the training set data in the vehicle appearance dataset to train the vehicle appearance defect detection network; The training of the vehicle appearance defect detection network comprises the following steps: The input data volume of the vehicle appearance defect detection network is set. According to the input data volume, the maximum length and maximum width of each batch of images input into the vehicle appearance defect detection network during the training process are determined, and black pixels are filled in all directions of each input image to make the length and width of all input images in the same batch consistent; The input image is input into the hourglass network, the hourglass network outputs an image of a region of interest corresponding to the input image, the image of the region of interest is regarded as a heat map of the input image, a threshold of the heat map is set, and the input image is cropped according to the set threshold to obtain an image that may have defects; Input the image that may have defects into a bidirectional multi-scale feature network, and the bidirectional multi-scale feature network outputs a multi-scale network downsampled 8 times feature map, a downsampled 16 times feature map, and a downsampled 32 times feature map of the input image in the step; The downsampled 8-fold feature map, the downsampled 16-fold feature map, and the downsampled 32-fold feature map of the multi-scale network are respectively input into three skip connection convolutional detection network branches, and the three skip connection convolutional detection network branches perform defect detection on the downsampled 8-fold feature map, the downsampled 16-fold feature map, and the downsampled 32-fold feature map according to the characteristics of the data in the vehicle appearance data set to obtain an image prediction box that may have defects, and the image prediction box that may have defects is a classification score that includes the coordinates of the center point of the prediction box and the length and width of the prediction box; A non-maximum suppression algorithm is adopted, and a non-maximum suppression algorithm threshold is set. The classification scores of the image prediction frames that may have defects are screened, and the image prediction frames whose classification scores are greater than the threshold are selected to obtain the final detected defect detection frames. A loss function is used to calculate the loss of the detected defect detection frames and the labeled frames, and the gradients are fed back to the vehicle appearance defect detection network to complete the training of the vehicle appearance defect detection network.
2. The vehicle appearance defect detection method according to claim 1, characterized in that: The construction of the vehicle appearance dataset includes: Collect appearance defect images of the vehicle to be inspected; The position of the vehicle appearance defect is marked on the vehicle appearance defect image to obtain a defect marking box, which includes the defect location and the defect, and the defect marking box is stored in a vehicle appearance dataset. The vehicle appearance dataset contains the vehicle appearance defect image and the normal vehicle appearance image, the coordinates of the center point in the defect marking box, the length and width of the marking box, and the marking box category.
3. The vehicle appearance defect detection method according to claim 1, characterized in that: The data augmentation and clustering method comprises: The vehicle appearance defect images and normal vehicle appearance images in the vehicle appearance dataset are randomly rotated, scaled, brightness changed and blurred to obtain the vehicle appearance defect images and normal vehicle appearance images after augmentation transformation, and the vehicle appearance defect images and normal vehicle appearance images after augmentation transformation are added to the vehicle appearance dataset; Processing the data set after the augmentation transformation, cropping the defect annotation frames whose area is less than 1% of the entire image area according to the defect position, and randomly pasting the cropped defects to other background areas without defects in the same image for multiple times to obtain multiple defect annotation frames, and storing the center point coordinates, length and width of the annotation frames, and the annotation frame categories of the multiple defect annotation frames in the vehicle appearance data set respectively; The ratios of the length and width of multiple defect annotation boxes in the vehicle appearance dataset to the length and width of the corresponding images are respectively counted; the multiple length-width ratios are clustered using the K-Means method to obtain nine pre-selected anchor boxes, and the nine pre-selected anchor boxes are sorted in sequence according to size; the three smallest boxes among the nine pre-selected anchor boxes after sorting are used as anchor boxes for feature maps downsampled 8 times, three medium-sized boxes are used as anchor boxes for feature maps downsampled 16 times, and the three largest boxes are used as anchor boxes for feature maps downsampled 32 times, so as to obtain the characteristics of the data in the vehicle appearance dataset.
4. The vehicle appearance defect detection method according to claim 1, characterized in that: The convolution layer in the feature extraction stage of the bidirectional multi-scale feature network adopts the LeakyRelu activation function, and performs the first feature map downsampling through spatial depth conversion, that is, converting a feature map into four images with a length and width of half the original feature map, thereby realizing feature map downsampling; The bidirectional multi-scale feature network uses the pyramid pooling method instead of ordinary pooling. The specific operation is: the feature channel is divided into four parts, and then spliced after passing through pooling layers with different pooling rates. The pooling rate is designed according to the actual situation of the network.
5. The vehicle appearance defect detection method according to claim 1, characterized in that: The detection output layer activation function of the skip connection convolutional detection network adopts the SWISH activation function; The skip connection convolution detection network uses the pyramid pooling method instead of ordinary pooling. The specific operation is to divide the feature channel into four parts, pass through pooling layers with different pooling rates, and then splice them. The pooling rate is designed according to the actual situation of the network.
6. The vehicle appearance defect detection method according to claim 1, characterized in that: The selected detection method includes: (1) When the vehicle appearance defect detection speed is greater than 30 frames per second, the vehicle appearance image to be inspected is scaled and input into the bidirectional multi-scale network in the vehicle appearance defect detection network; when the vehicle appearance defect detection speed is less than or equal to 30 frames per second, the vehicle appearance image to be inspected is input into the hourglass network in the vehicle appearance defect detection network for cropping, and the cropped image is input into the bidirectional multi-scale network; (2) Calculate the minimum scaling size required for downsampling the appearance image of the vehicle to be inspected by 32 times, then fill black pixels around the appearance image of the vehicle to be inspected to make the appearance image of the vehicle to be inspected reach the calculated size, and use the appearance image of the vehicle to be inspected as the input of the vehicle appearance detection network to obtain the final detected defect detection frame; (3) The non-maximum suppression method is used to select the prediction boxes with a classification score of more than 50% from the final detected defect detection boxes as defect detection boxes, thereby realizing the detection of vehicle appearance defects based on convolutional neural networks.
7. A vehicle appearance defect detection device, characterized in that: include: An image acquisition module is used to acquire vehicle appearance defect images and normal vehicle appearance images, and to annotate the acquired vehicle appearance defect images and normal vehicle appearance images respectively, so as to construct a vehicle appearance data set; An image processing module, used for performing data preprocessing and data set division on the vehicle appearance data set, using data augmentation and clustering methods to increase the data volume of the vehicle appearance data set, and summarizing data characteristics; The network construction and training module is used to build a vehicle appearance defect detection network and use the preprocessed parameters and data for training; A defect detection module is used to select a detection method according to demand, import the image to be detected into the trained vehicle appearance defect detection network, output the defect area and defect type, and realize vehicle appearance defect detection based on convolutional neural network; The network construction and training module is also used to: Construct a vehicle appearance defect detection network, including an hourglass network, a bidirectional multi-scale feature network, and three skip-connected convolutional detection network branches; 10% of the data in the vehicle appearance dataset is used as a test set, and 90% is used as a training set; Use the training set data in the vehicle appearance dataset to train the vehicle appearance defect detection network; The training of the vehicle appearance defect detection network includes: The input data volume of the vehicle appearance defect detection network is set. According to the input data volume, the maximum length and maximum width of each batch of images input into the vehicle appearance defect detection network during the training process are determined, and black pixels are filled in all directions of each input image to make the length and width of all input images in the same batch consistent; The input image is input into the hourglass network, the hourglass network outputs an image of a region of interest corresponding to the input image, the image of the region of interest is regarded as a heat map of the input image, a threshold of the heat map is set, and the input image is cropped according to the set threshold to obtain an image that may have defects; Input the image that may have defects into a bidirectional multi-scale feature network, and the bidirectional multi-scale feature network outputs a multi-scale network downsampled 8 times feature map, a downsampled 16 times feature map, and a downsampled 32 times feature map of the input image in the step; The downsampled 8-fold feature map, the downsampled 16-fold feature map, and the downsampled 32-fold feature map of the multi-scale network are respectively input into three skip connection convolutional detection network branches, and the three skip connection convolutional detection network branches perform defect detection on the downsampled 8-fold feature map, the downsampled 16-fold feature map, and the downsampled 32-fold feature map according to the characteristics of the data in the vehicle appearance data set to obtain an image prediction box that may have defects, and the image prediction box that may have defects is a classification score that includes the coordinates of the center point of the prediction box and the length and width of the prediction box; A non-maximum suppression algorithm is adopted, and a non-maximum suppression algorithm threshold is set. The classification scores of the image prediction frames that may have defects are screened, and the image prediction frames whose classification scores are greater than the threshold are selected to obtain the final detected defect detection frames. A loss function is used to calculate the loss of the detected defect detection frames and the labeled frames, and the gradients are fed back to the vehicle appearance defect detection network to complete the training of the vehicle appearance defect detection network.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to enable the computer to execute the vehicle appearance defect detection method described in any one of claims 1-6.