A data enhancement method for coating defect detection

Abstract

The present invention discloses a data enhancement method and system applied in the field of coating defect detection, which is mainly aimed at coating defect data sets with small sample data volume, uneven distribution of proportions of various categories, and a large number of small samples that are densely clustered, including: 1. Producing a feature map database based on the original data set; 2. Based on the prior accuracy and quantity of each category of the original image, randomly extracting corresponding feature maps from the feature map database and adaptively weighted mapping to the original image; 3. Transforming, combining and splicing with multiple other original images with a certain probability into a new image. The new image generated by the present invention has a low correlation with the original image, and can dynamically adjust hyperparameters and functions according to different complex application environments, enrich sample features and quantity, and avoid the overfitting problem that may be caused by traditional data enhancement. The ablation experiment shows that the model trained by the data enhancement has better recognition accuracy and generalization, and the method does not require manual secondary labeling.

Description

Technical Field

[0001] The present invention relates to the field of neural network model image data enhancement methods, and in particular to a data enhancement method applied to the field of coating defect detection. Background Art

[0002] In the production process, spraying is often the last step in product processing. Spraying has three main functions: 1. It can protect objects from being eroded by various media such as air and water, and extend their service life; 2. Spraying special materials on the surface of objects can be used for camouflage, fire prevention, waterproofing, and anti-fouling; 3. It enhances the gloss and color of the surface of objects to enhance beauty and play a decorative role. However, in the actual spraying process, due to factors such as paint quality, spraying technology, and workshop environment, defects may occur in the surface coating of the workpiece during and after the spraying process. These defects may have an important impact on the appearance, service life, and safety of the workpiece. Therefore, it is very necessary to detect product coating defects.

[0003] At present, most companies still use manual or traditional machine vision methods to detect coating defects. Manual detection methods are inefficient, inaccurate, and easily affected by human experience and subjective factors. Traditional machine vision requires manual design of features. Although it effectively filters background noise to a certain extent, the algorithm has poor generalization and performs poorly in complex industrial environments. The defect detection algorithm based on deep learning uses huge training data and automatically extracts target-rich features through operations such as convolution, weight sharing, and pooling, making the algorithm more generalizable and expressive. It has been proven in applications in different fields that it can handle more complex classification problems.

[0004] For coating defect detection algorithms based on deep learning, the quality of the data set plays a crucial role in the generalization and recognition accuracy of the model. However, in practice, it is common that the amount of defect sample data is insufficient or the quality is not high. It is necessary to perform data enhancement on the samples based on certain methods to improve the overall quality of the samples.

[0005] The current coating defect dataset has the following characteristics: 1. The amount of real defect sample data is small and difficult to collect; 2. It is easy to reflect light when exposed to the natural environment, and the background is complex and changeable; 3. The morphology of different samples is quite different, and the number of samples is seriously unbalanced; 4. There are a large number of small targets, which are densely clustered. These characteristics hinder the improvement of model recognition accuracy, and some existing data enhancement methods are not applicable or only solve part of the problem.

[0006] For example, the publication number is CN113793279A, and the name is "A data enhancement method for traffic sign target detection". This method crops a small sample and directly embeds it into the matched image. It does not consider the phenomenon that the original positive sample in the matched image is partially blocked after embedding, causing some features to be lost and the position of the annotation box to be shifted. In addition, this method does not have a set of principles for judging and selecting the category, quantity and embedding position of the cropped images, which may result in class imbalance and the enhanced image not conforming to the actual situation.

[0007] The publication number is CN111583183A, and the name is "A data enhancement method and system for PCB board image defect detection". This method only performs random cropping and scaling operations on the original image to expand the number of small samples. The new image generated is highly correlated with the original image, and the improvement on the model detection effect is very small, and it is easily affected by the external environment.

[0008] The publication number is CN112270653A, and the name is "A data enhancement method for image sample imbalance". This method only solves the problem of partial sample imbalance, but it does not solve the problem of a large number of small targets and dense clustering in most defect detection data sets.

[0009] Therefore, in the field of coating defect detection, a new data enhancement method and system is needed to uniformly solve the shortcomings of existing data enhancement methods and improve the overall quality of coating defect datasets, thereby further improving the accuracy and generalization of model detection. Summary of the invention

[0010] The purpose of the present invention is to provide a data enhancement method applied to the field of coating defect detection, so as to solve the shortcomings of the data enhancement method in the prior art and improve the accuracy and robustness of the coating defect detection algorithm.

[0011] In order to achieve the above object, the technical solution adopted by the present invention is:

[0012] A data enhancement method applied in the field of coating defect detection comprises the following steps:

[0013] Step 1: obtain multiple pictures, each picture showing at least one type of coating defect, and manually mark each picture with a corresponding label according to at least one coating defect category corresponding to the picture;

[0014] Step 2: divide some of the multiple images marked with labels in step 1 into a pre-training set, and divide the remaining images into a test set, and process the images in the pre-training set and the test set respectively, and convert them into the data format required for deep learning network model training;

[0015] Step 3: pre-train the deep learning network model using the image data of the pre-training set obtained in step 2, and test the deep learning network model using the image data of the test set obtained in step 2, and obtain the prior recognition accuracy of the deep learning network model for various coating defect categories according to the test results;

[0016] Step 4: Determine the position of the corresponding coating defect part from each picture in the pre-training set, separate all the coating defect parts as feature pictures, and establish a coating defect feature map database according to the category of coating defects corresponding to the feature pictures, wherein the coating defect feature map database includes the feature pictures corresponding to each picture in the pre-training set and classifies the feature pictures of each picture according to the category of coating defects;

[0017] Step 5: Randomly select a picture from the pre-training set as the original picture to be enhanced, perform a first random transformation on the original picture to be enhanced to obtain a transformed picture, and obtain a label for the first randomly transformed picture based on the matrix transformation and image processing technology used in the first random transformation process. The first random transformation is used to reduce the correlation between the final enhanced picture and the original picture;

[0018] Step 6: Based on the coating defect categories contained in the original image to be enhanced, the prior recognition accuracy of the coating defect categories contained in the original image to be enhanced in the deep learning network model, and the number of each coating defect category contained in the original image to be enhanced, the number of feature images to be mapped of each category contained in the original image to be enhanced is randomly selected from the coating defect feature image database according to the following formula:

[0019]

[0020] Among them, p i is the prior recognition accuracy of the coating defect category contained in the original image to be enhanced in the deep learning network model, i = 1, 2, 3, 4...n, n is the total number of coating defect categories contained in the original image to be enhanced, m is the total number of feature images contained in the original image to be enhanced, and v is the hyperparameter coefficient that can dynamically adjust the number of a specific category randomly selected to alleviate the class imbalance problem;

[0021] Then, a second random transformation is performed on each randomly selected feature image to obtain a transformed feature image. The second random transformation is performed to reduce the correlation between the final enhanced image and the original image;

[0022] Step 7: The feature image after the second random transformation is mapped to the image after the first random transformation through adaptive weighted mapping, the corresponding annotation label of the original image to be enhanced after the first random transformation is updated, and the mapped original image and the corresponding annotation label are subjected to a third random transformation to obtain the transformed image, and the annotation label of the image after the third random transformation is obtained according to the matrix transformation and image processing technology used in the third random transformation process. The third random transformation is used to reduce the correlation between the final enhanced image and the original image;

[0023] Step 8, randomly generate a random number from 1 to 9. If the generated random number is from 7 to 9, randomly select multiple different pictures and corresponding annotation labels from the pre-training set, splice the selected pictures with the pictures after the third random transformation to obtain the final data-enhanced picture, and obtain the annotation label of the final data-enhanced picture according to the matrix transformation and image processing technology used in the splicing process, add the final data-enhanced picture and the corresponding annotation label to the new training set for subsequent training of the deep learning network model. Otherwise, the pictures and corresponding labels after the third random transformation are used as the final data-enhanced pictures and corresponding annotation labels, and added to the new training set for subsequent training of the deep learning network model.

[0024] In the further step 2, the pre-training set and the test set are divided into a ratio of 6:4.

[0025] Furthermore, the deep learning network model described in steps 2, 3, and 6 is a YOLOv5 model.

[0026] In a further step 4, the coordinates of the coating defect parts contained in each picture of the pre-training set are determined respectively, and the categories of the coating defects contained in each picture are counted. Then, all the coating defect parts are cropped out from the picture according to the coordinates as feature pictures, and then the coating defects of each picture are classified based on the category, thereby obtaining a coating defect feature map database.

[0027] In a further step 5, the first random transformation includes random scaling of the image size, random mirroring, random center point movement and random rotation operations.

[0028] In the further step 6, firstly, a statistical analysis is performed on the background environment and feature images of the original image to be enhanced, and some feature images that do not conform to the scene of the original image to be enhanced when the feature images are randomly selected are filtered out through the coating defect categories existing in the background of the original image to be enhanced, so that the data enhancement conforms to the actual situation; then, multiple feature images are randomly selected from the coating defect feature image database according to the formula.

[0029] In a further step 6, the second random transformation includes random image scaling, random mirroring and random rotation operations.

[0030] Furthermore, the adaptive weighted mapping process in step 7 is as follows:

[0031] (7.1) Let the width and height of the original image to be enhanced be w1 and h1, and the width and height of the feature image be f_w and f_h;

[0032] (7.2) Calculate f_w / 2 and f_h / 2. If f_w / 2≥w1-f_w / 2 or f_h / 2≥h1-f_h / 2 is satisfied, it is determined that the feature image is not suitable for the original image to be enhanced, and the feature image is discarded.

[0033] (7.3) Calculate f_w / 2 and f_h / 2. If f_w / 2≥w1-f_w / 2 or f_h / 2≥h1-f_h / 2 is not satisfied, then randomly select the center point of the feature image Fea_c=[random(f_w / 2,w1-f_w / 2), random(f_h / 2,h1-f_h / 2)] in the image after the first random transformation, and obtain the prior annotation box box=[xmin,ymin,xmax,ymax] of the feature image in the image after the first random transformation;

[0034] Then, the intersection area coordinates inter_box and the intersection area of ​​the prior annotation box and the annotation box in the image after the first random transformation are calculated. When the ratio of the intersection area to the area of ​​the prior annotation box or the ratio of the intersection area to the area of ​​the annotation box in the image after the first random transformation is greater than the initial set threshold conf, it is determined that the position of the feature image in the image after the first random transformation is inappropriate, and this step (7.3) is repeated, otherwise proceed to the next step;

[0035] (7.4) To prevent the feature image from being mapped to the image after the first random transformation, causing some features of the image after the first random transformation to be lost and the position of the annotation box to be offset, calculate the area of ​​the intersection of the prior annotation box box obtained in step (7.3) and each annotation box in the image after the first random transformation;

[0036] If the area of ​​the intersection region is greater than 0, the intersection image inter_img1 in the image to be subjected to the first random transformation is cropped according to the intersection region coordinates inter_box, and then the feature image is mapped to the image after the first random transformation to obtain the mapped original image to be enhanced img1;

[0037] Then, the image inter_img2 in img1 is cropped again according to the intersection area coordinates inter_box, and the weighted fused image fuse_img of the intersection area is obtained by the following formula:

[0038] fuse_img=inter_img1×0.5+inter_img2×0.5

[0039] Map the weighted fused image fuse_img to img1 according to the intersection area coordinates inter_box, and add the prior annotation box box to the label after the first random transformation.

[0040] In a further step (7.3), if the ratio of the intersection area to the box area or the ratio of the intersection area to the area of ​​the corresponding image annotation box after the first random transformation is still greater than the initial set threshold conf when the process is repeated more than 1000 times, it is judged that the feature image is not suitable for the image after the first random transformation, the feature image is discarded, and steps (7.2) and (7.3) are repeated.

[0041] In a further step 7, the third random transformation includes random brightness, random Gaussian blur and random noise operations.

[0042] Furthermore, in step (8), if probabilistic splicing occurs, a blank image is first constructed, the size of which is twice the size of the input image of the deep learning network model. Then, the coordinates of the splicing points are randomly selected in the blank image to divide the blank image into 4 blocks. The size of the image to be spliced ​​is adjusted and randomly mapped to the 4 blank images. Finally, the mapped image is scaled and offset, and the annotation label information is adjusted to obtain the spliced ​​image and the corresponding label.

[0043] The method of the present invention can be applied to coating defect detection tasks under different complex backgrounds, and does not require manual secondary annotation. It can be easily inserted into the training code to achieve online data enhancement. Compared with the prior art, the advantages of the present invention are:

[0044] (1) The image enhanced by the data of the present invention has less correlation with the original image;

[0045] (2) The present invention designs a method for preparing a feature map database and extracting feature maps, and dynamically adjusts various hyperparameters and functions according to different background environments to enrich the number of features and avoid the overfitting problem after traditional data enhancement;

[0046] (3) Compared with the cropping and pasting feature maps in other data enhancement methods, the present invention solves the problems of partial feature loss and original annotation box position offset in existing data enhancement methods through adaptive feature map screening, setting threshold conf and intersection area image weighted mapping method, and achieves better feature fusion effect;

[0047] (4) The data-enhanced images do not require manual secondary annotation and can be easily inserted into the training code for online data enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flow chart of a method according to an embodiment of the present invention.

[0049] Figure 2 This is the coating defect dataset preparation process of the embodiment of the present invention.

[0050] Figure 3 It is a schematic diagram of the effect after adaptive weighted fusion according to an embodiment of the present invention.

[0051] Figure 4 It is a schematic diagram of the effects before and after data enhancement is performed on the original image to be enhanced according to an embodiment of the present invention.

[0052] Figure 5 It is a graph of ablation experiment results on a coating defect dataset according to an embodiment of the present invention.

[0053] Figure 6 4 is a graph showing the ablation experiment results of an embodiment of the present invention on a VOC dataset. DETAILED DESCRIPTION

[0054] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0055] like Figure 1 As shown, the method of this embodiment includes the following steps:

[0056] Step 1: Figure 2 As shown, in order to increase the number of samples in the original data set, the causes of coating defects are first analyzed, and coating defect samples are artificially produced according to the causes of coating defects.

[0057] Then, a 20-megapixel industrial camera is used (a larger pixel camera is used to prevent the loss of features of some tiny defects in subsequent detection), and images of the coating defect samples are collected under different background environments and lighting, thereby obtaining multiple images, each of which shows at least one category of coating defects contained in the corresponding coating defect sample. The image format is .jpg format and is annotated using labelImg software to mark each image with a corresponding label according to its corresponding coating defect category, and the label reflects the coating defect information.

[0058] Step 2: Divide the multiple images in step 1 into a pre-training set and a test set in a ratio of 6:4. The pre-training set will be used for subsequent data enhancement, while the test set will be used to test and evaluate the trained deep learning network model. The images in the pre-training set and the test set will be processed separately and converted into the data format required for deep learning network model training.

[0059] Step 3: Use the image data of the pre-training set obtained in step 2 to pre-train the deep learning network model, and use the image data of the test set obtained in step 2 to test the deep learning network model. According to the test results, the prior recognition accuracy of the deep learning network model for various coating defect categories under the 0.5mAP index is obtained.

[0060] In this embodiment, the YOLOv5 model is used as the deep learning network training model.

[0061] Since the recognition accuracy of coating defect categories with less data is not necessarily lower, the accuracy of sample recognition by the deep learning network model is related to factors such as the number of samples, feature complexity, and background environment, and it is not enough to simply consider balancing the number of categories during data enhancement. Therefore, compared with other data enhancement methods that directly expand categories with less data in order to balance the number of categories, this embodiment uses the prior recognition accuracy of each coating defect category as one of the principles for subsequent sample expansion.

[0062] Step 4: Statistically analyze the coating defect categories and the number of each coating defect category in the pre-training set, and create folders named after each coating defect category.

[0063] According to the coating defect category in the annotation label of each picture in the pre-training set, the coordinate information of the coating defect part in the corresponding picture (including the normalized x-coordinate of the center point of the annotation box, the normalized y-coordinate of the center point of the annotation box, the normalized width of the annotation box, and the normalized height of the annotation box), each picture is cropped to separate the coating defect part as the feature picture of the corresponding picture, and the feature picture is saved in the corresponding coating defect category folder, thereby establishing a coating defect feature map database. Therefore, the coating defect feature map database contains the feature picture corresponding to each picture in the pre-training set, and classifies the feature pictures of each picture according to the category of the coating defect.

[0064] Step 5: Randomly select a picture from the pre-training set as the original picture to be enhanced, perform a first random transformation on the original picture to be enhanced to obtain a transformed picture, and obtain a label for the first randomly transformed picture based on the matrix transformation and image processing technology used in the first random transformation process. The first random transformation reduces the correlation between the final enhanced picture and the original picture. The specific process is as follows:

[0065] Randomly extract an original image img0 to be enhanced and the corresponding annotation label labels from the pre-training set. The original image img0 to be enhanced is the array after OpenCV reads the image, labels = [m, 5], where m is the total number of positive sample annotation boxes, and the number 5 refers to the coating defect category, the x-coordinate of the center point of the annotation box, the y-coordinate of the center point of the annotation box, the width of the annotation box, and the height of the annotation box, a total of 5 attributes.

[0066] The first random transformation is used to obtain new images img0 and labels, wherein the first random transformation includes random image scaling, random mirroring, random center point movement, and random rotation operations. Specifically, the value ranges of the first random transformation hyperparameters (including random image scaling ratio, random flipping method, random center point movement range, and random rotation angle) are shown in Table 1:

[0067] Table 1 The range of values ​​of the first random transformation hyperparameters

[0068] Transformation function Probability of use Specific value range of hyperparameters Randomly scale image size 1 0.6～1.2 Random Flip 1 Random horizontal and vertical flip Random center point movement 1 (-400，400) Random rotation 1 (90°,180°,270°)

[0069] The hyperparameters and transformation functions in the first random transformation can be dynamically adjusted to meet the data enhancement requirements in different background environments.

[0070] Step 6: To solve the problems of small amount of data in each category, uneven distribution of proportions, and a large number of small targets densely clustered in the pre-training set, this embodiment comprehensively considers the prior recognition accuracy, quantity, and morphological size of each category, and based on the coating defect categories contained in the original image to be enhanced, the recognition accuracy of the coating defect categories contained in the original image to be enhanced in the deep learning network model, and the number of coating defect categories contained in the original image to be enhanced, multiple feature images of each category contained in the original image to be enhanced are randomly selected from the coating defect feature image database to simulate the sample distribution in the real scene, thereby solving the problem of the shortcomings of the existing data enhancement method. The specific process is as follows

[0071] (6.1) Statistical analysis is performed on the coating defect categories and the number of feature images corresponding to the background environment and feature images in the original image img0 to be enhanced. The coating defect categories existing in the original image img0 to be enhanced are used to filter out features that should not exist in some specific environments (e.g., in the scene of the living room of a house, airplanes and trains in real life should not appear), thereby determining the target sample categories that need to be enhanced for the original image img0 to be enhanced, so that the data enhancement conforms to the actual situation.

[0072] (6.2) Assume that the coating defect category in the original image img0 to be enhanced is c = [c1, c2, …, c n ], the prior recognition accuracy of each coating defect category in the deep learning network model pre-training in step 3 is The number of feature images of each category contained in the original image to be enhanced is determined by randomly selecting from the coating defect feature image database:

[0073]

[0074] Among them, p iis the prior recognition accuracy of the coating defect category contained in the original image to be enhanced in the deep learning network model, i = 1, 2, 3, 4...n, n is the total number of coating defect categories contained in the original image to be enhanced, m is the total number of coating defect categories contained in the original image to be enhanced, and v is the hyper-parameter coefficient that can dynamically adjust the number of a specific category randomly selected to alleviate the class imbalance problem.

[0075] (6.3) Use OpenCV to read each feature image and create a feature image list array FeaImgs. Perform a second random transformation on each feature image array in FeaImgs to obtain the transformed image.

[0076] The second random transformation includes random scaling, random flipping and random rotation operations. Specifically, the value ranges of the second random transformation hyperparameters (including the random scaling ratio, random flipping method and random rotation angle) are shown in Table 2:

[0077] Table 2 Second random transformation hyperparameter value range

[0078] Transformation function Probability of use Specific value range of hyperparameters Randomly scale image size 1 0.6～1.2 Random Flip 1 Random horizontal and vertical flip Random rotation 1 (90°,180°,270°)

[0079] The hyperparameters and transformation functions in the second random transformation can be dynamically adjusted according to the characteristics and requirements of different feature images to meet the data enhancement requirements under different background environments.

[0080] Step 7: The feature image after the second random transformation is mapped to the image after the first random transformation through adaptive weighted mapping, the corresponding annotation label of the original image to be enhanced after the first random transformation is updated, and the mapped image is subjected to a third random transformation to obtain the transformed image, and the annotation label of the image after the third random transformation is obtained according to the matrix transformation and image processing technology used in the third random transformation process.

[0081] In this embodiment, in order to better map and fuse the feature image with the original image to be enhanced img0, and solve the problems of partial feature loss and position offset of the annotation box in the prior art method, this embodiment adopts an adaptive weighted mapping fusion strategy, and the specific process is as follows:

[0082] (7.1) Let the width and height of the original image img0 to be enhanced be w1 and h1, extract a feature image FeaImg to be mapped from FeaImgs, and the width and height of the feature image FeaImg are f_w and f_h;

[0083] (7.2) Calculate f_w / 2 and f_h / 2. If f_w / 2≥w1-f_w / 2 or f_h / 2≥h1-f_h / 2 is satisfied, it is determined that the feature image is not suitable for the original image to be enhanced, and the feature image is discarded.

[0084] (7.3) Calculate f_w / 2 and f_h / 2. If f_w / 2≥w1-f_w / 2 or f_h / 2≥h1-f_h / 2 is not satisfied, then randomly select the center point Fea_c=[random(f_w / 2,w1-f_w / 2), random(f_h / 2,h1-f_h / 2)] of the feature image FeaImg in the image after the first random transformation, and obtain the prior annotation box box=[xmin,ymin,xmax,ymax] of the feature image FeaImg in the image after the first random transformation;

[0085] Then, the intersection area coordinates inter_box (inter_box = [in_xmin, in_ymin, in_xmax, in_ymax]) and the intersection area of ​​the prior annotation box and each annotation box in the image after the first random transformation are calculated one by one. When the ratio of the intersection area to the area of ​​the prior annotation box or the ratio of the intersection area to the area of ​​the annotation box in the image after the first random transformation is greater than the initial set threshold conf (generally set to 0.35, which can be adjusted according to actual conditions), it is judged that the position of the feature image FeaImg in the image after the first random transformation is inappropriate, and this step (7.3) is repeated, otherwise proceed to the next step;

[0086] (7.4) To prevent the feature image FeaImg from being mapped to the image after the first random transformation, causing some features in the image after the first random transformation to be lost and the position of the annotation box to be offset, calculate the area of ​​the intersection of the prior annotation box box obtained in step (7.3) and each annotation box in the image after the first random transformation;

[0087] If the area of ​​the intersection region is greater than 0, the intersection image inter_img1=img0[in_ymin:in_ymax, in_xmin:in_xmax] in the image after the first random transformation is cropped according to the intersection region coordinates inter_box, and the feature image FeaImg is mapped to the image after the first random transformation to obtain the mapped original image to be enhanced img1=img0[ymin:ymax,xmin:xmax];

[0088] Then, the image inter_img2=img1[in_ymin:in_ymax, in_xmin:in_xmax] in the original image img1 to be enhanced after mapping is cropped again according to the intersection area coordinates inter_box, and the weighted fused image fuse_img of the intersection area is obtained by the following formula:

[0089] fuse_img=inter_img1×0.5+inter_img2×0.5

[0090] Map the weighted fused image fuse_img to img1 according to the intersection area coordinates inter_box, and add the prior annotation box box to the corresponding annotation label labels of the image after the first random transformation.

[0091] Compared with the general fusion method, the adaptive weighted mapping fusion effect of the present invention is as follows: Figure 3 As shown. Figure 3 It can be seen that the general mapping method directly maps the feature image to the original image. When the feature image and the annotation box in the original image to be mapped have an intersection, the feature image may cover some features in the original image to be mapped, resulting in partial feature loss and position shift of the annotation box, thereby affecting the recognition and positioning effect of the model. The intersection area weighted mapping method proposed by the present invention can solve these problems well.

[0092] Finally, the mapped original image img1 to be enhanced is subjected to a third random transformation, and the third random transformation includes random brightness, random Gaussian blur and random noise operations. Specifically, the value range of the third random transformation hyperparameter (including random brightness range, random Gaussian blur filter size and random noise ratio) is shown in Table 3:

[0093] Table 3. The value range of the third random transformation hyperparameter

[0094]

[0095]

[0096] Step 8, the image img1 after the third random transformation has a probability of 1 / 3 to be spliced ​​with multiple other images randomly selected from the pre-training set. The specific operation includes randomly generating a random number from 1 to 9. If the generated random number is between 7 and 9, multiple different images and corresponding annotation labels are randomly selected from the pre-training set, and the selected multiple images are spliced ​​with the image after the third random transformation to obtain the final data-enhanced image, and the annotation label of the final data-enhanced image is obtained according to the matrix transformation and image processing technology used in the splicing process, and the final data-enhanced image and the corresponding annotation label are added to the new training set for subsequent training of the deep learning network model. Otherwise, the image after the third random transformation and the corresponding label are used as the final data-enhanced image and the corresponding label, and added to the new training set for subsequent training of the deep learning network model.

[0097] The comparison effect between the image after the final data enhancement of the present invention and the image before the data enhancement is as follows: Figure 4 As shown. Figure 4It can be seen that compared with the existing data enhancement technology, the image after data enhancement by the present invention has less correlation with the original image, and the number of samples and the size of the morphology are richer, which avoids the overfitting problem after traditional data enhancement. The problem of imbalanced data sets can also be solved by adjusting the overfitting variables in the data enhancement process.

[0098] The method of the present invention verifies:

[0099] In order to verify the effectiveness of the method of the present invention, the 1300 coating defect images collected, about 14900 defect targets, were divided into 780 pre-training sets and 520 validation sets by 6:4. Then the pre-training set was enhanced by the method of the present invention and expanded to 1400 images. The ablation experiment was carried out with the YOLOv5 model as the baseline. The experimental results are shown in the figure. Figure 5 As shown. Figure 5 It can be seen that the YOLOv5 model is used as the baseline for model training and testing. Compared with the traditional data enhancement method and other popular data enhancement methods, the recognition accuracy of the coating defect detection of the method of the present invention is higher, reaching 95.6%. In order to verify the versatility of the method of the present invention, a comparative experiment was conducted on the public data set VOC. The experimental results are as follows: Figure 6 As shown, the results show that the method of the present invention is also effective for other detection tasks, thus further proving the effectiveness of the method of the present invention.

[0100] The present invention uses Python programming language and OpenCV (OpenCV is an open source cross-platform computer vision and machine learning software library) to implement the algorithm process, and encapsulates the program of the algorithm of the present invention for easy insertion into the training code.

[0101] In summary, the present invention proposes a data enhancement method and system for the field of coating defect detection. Through the production of feature map database, feature map extraction strategy and adaptive weighted mapping strategy, the newly generated image has a low correlation with the original image, which improves the overall quality of the existing coating defect data set, and solves the problem of partial feature loss and original annotation box offset in the feature fusion process of other data enhancement methods. Finally, based on the YOLOv5 algorithm, ablation experiments were performed on the coating defect data set and VOC data set after data enhancement of the present invention. The results show that the data enhancement method after the present invention improves the accuracy and generalization of the coating defect detection model, and the method of the present invention is also effective for other data sets.

[0102] The embodiments described in the present invention are merely descriptions of the preferred implementation modes of the present invention, and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made to the technical solutions of the present invention by engineers and technicians in this field should fall within the protection scope of the present invention. The technical contents for which protection is sought in the present invention have all been recorded in the claims.

Claims

1. A data enhancement method applied in the field of coating defect detection, characterized in that: The following steps are involved: Step 1: obtain multiple pictures, each picture showing at least one type of coating defect, and manually mark each picture with a corresponding label according to at least one coating defect category corresponding to the picture; Step 2: divide some of the multiple images marked with labels in step 1 into a pre-training set, and divide the remaining images into a test set, and process the images in the pre-training set and the test set respectively, and convert them into the data format required for deep learning network model training; Step 3: pre-train the deep learning network model using the image data of the pre-training set obtained in step 2, and test the deep learning network model using the image data of the test set obtained in step 2, and obtain the prior recognition accuracy of the deep learning network model for various coating defect categories according to the test results; Step 4: Determine the position of the corresponding coating defect part in each picture of the pre-training set, and separate all the coating defect parts as feature pictures. According to the category of coating defects corresponding to the feature pictures, establish a coating defect feature map database, wherein the coating defect feature map database contains the feature pictures corresponding to each picture of the pre-training set, and classifies the feature pictures of each picture according to the category of coating defects; Step 5: randomly select a picture from the pre-training set as the original picture to be enhanced, perform a first random transformation on the original picture to be enhanced to obtain a transformed picture, and obtain a label of the picture after the first random transformation according to the matrix transformation and image processing technology used in the first random transformation process, and reduce the correlation between the final enhanced picture and the original picture through the first random transformation; Step 6: Based on the coating defect categories contained in the original image to be enhanced, the prior recognition accuracy of the coating defect categories contained in the original image to be enhanced in the deep learning network model, and the number of each coating defect category contained in the original image to be enhanced, the number of feature images to be mapped of each category contained in the original image to be enhanced is randomly selected from the coating defect feature image database according to the following formula: Among them, p i is the prior recognition accuracy of the coating defect category contained in the original image to be enhanced in the deep learning network model, i = 1, 2, 3, 4...n, n is the total number of coating defect categories contained in the original image to be enhanced, m is the total number of feature images contained in the original image to be enhanced, and v is the hyperparameter coefficient that can dynamically adjust the number of a specific category randomly selected to alleviate the class imbalance problem; Then, a second random transformation is performed on each randomly selected feature image to obtain a transformed feature image, and the correlation between the final enhanced image and the original image is reduced by the second random transformation; Step 7: The feature image after the second random transformation is mapped to the image after the first random transformation through adaptive weighted mapping, the corresponding annotation label of the original image to be enhanced after the first random transformation is updated, and the mapped original image and the corresponding annotation label are subjected to a third random transformation to obtain a transformed image, and the annotation label of the image after the third random transformation is obtained according to the matrix transformation and image processing technology used in the third random transformation process, and the correlation between the final enhanced image and the original image is reduced through the third random transformation; Step 8. Randomly generate a random number from 1 to 9. If the generated random number is from 7 to 9, randomly select multiple different pictures and corresponding annotation labels from the pre-training set, splice the selected multiple pictures with the picture after the third random transformation to obtain a final data-enhanced picture, and obtain the annotation label of the final data-enhanced picture according to the matrix transformation and image processing technology used in the splicing process, add the final data-enhanced picture and the corresponding annotation label to the new training set for subsequent training of the deep learning network model, otherwise, the picture after the third random transformation and the corresponding label are used as the final data-enhanced picture and the corresponding annotation label, and added to the new training set for subsequent training of the deep learning network model.

2. A data enhancement method applied to the field of coating defect detection according to claim 1, characterized in that: In step 2, the pre-training set and the test set are divided into a ratio of 6:

4.

3. The data enhancement method applied to the field of coating defect detection according to claim 1, characterized in that: The deep learning network model described in steps 2, 3, and 6 is the YOLOv5 model.

4. The data enhancement method applied to the field of coating defect detection according to claim 1, characterized in that: In step 4, the coordinates of the coating defect parts contained in each picture of the pre-training set are determined respectively, and the categories of the coating defects contained in each picture are counted. Then, all the coating defect parts are cropped out from the picture according to the coordinates as feature pictures, and then the coating defects of each picture are classified based on the category, thereby obtaining a coating defect feature map database.

5. The data enhancement method applied to the field of coating defect detection according to claim 1, characterized in that: In step 5, the first random transformation includes random scaling of the image size, random mirroring, random center point movement, and random rotation operations.

6. The data enhancement method applied to the field of coating defect detection according to claim 1, characterized in that: In step 6, firstly, a statistical analysis is performed on the background environment and feature images in the original image to be enhanced, and some feature images that do not conform to the scene of the original image to be enhanced when the feature images are randomly selected are filtered out through the coating defect categories existing in the background of the original image to be enhanced, so that the data enhancement conforms to the actual situation; then, multiple feature images of each corresponding category are randomly selected from the coating defect feature image database according to the formula.

7. The data enhancement method applied to the field of coating defect detection according to claim 1, characterized in that: In step 6, the second random transformation includes random image scaling, random flipping, and random rotation operations.

8. The data enhancement method applied to the field of coating defect detection according to claim 1, characterized in that: The adaptive weighted mapping process described in step 7 is as follows: (7.1) Let the width and height of the original image to be enhanced be w1 and h1, and the width and height of the feature image be f_w and f_h; (7.2) Calculate f_w / 2 and f_h / 2. If f_w / 2≥w1-f_w / 2 or f_h / 2≥h1-f_h / 2 is satisfied, it is determined that the feature image is not suitable for the original image to be enhanced, and the feature image is discarded. (7.3) Calculate f_w / 2 and f_h / 2. If f_w / 2≥w1-f_w / 2 or f_h / 2≥h1-f_h / 2 is not satisfied, then randomly select the center point of the feature image Fea_c=[random(f_w / 2,w1-f_w / 2), random(f_h / 2,h1-f_h / 2)] in the image after the first random transformation, and obtain the prior annotation box box=[xmin,ymin,xmax,ymax] of the feature image in the image after the first random transformation; Then, the intersection area coordinates inter_box and the intersection area of ​​the prior annotation box and the annotation box in the image after the first random transformation are calculated. When the ratio of the intersection area to the area of ​​the prior annotation box or the ratio of the intersection area to the area of ​​the annotation box in the image after the first random transformation is greater than the initial set threshold conf, it is determined that the position of the feature image in the image after the first random transformation is inappropriate, and this step (7.3) is repeated, otherwise proceed to the next step; (7.4) To prevent the feature image from being mapped to the image after the first random transformation, causing some features in the image after the first random transformation to be lost and the position of the annotation box to be offset, calculate the area of ​​the intersection of the prior annotation box box obtained in step (7.3) and each annotation box in the image after the first random transformation; If the area of ​​the intersection region is greater than 0, the intersection image inter_img1 in the image after the first random transformation is cropped according to the intersection region coordinates inter_box, and then the feature image is mapped to the image after the first random transformation to obtain the mapped original image to be enhanced img1; Then, the image inter_img2 in img1 is cropped again according to the intersection area coordinates inter_box, and the weighted fused image fuse_img of the intersection area is obtained by the following formula: fuse_img=inter_img1×0.5+inter_img2×0.5 The weighted fused image fuse_img is mapped to img1 according to the intersection area coordinates inter_box, and the prior annotation box is added to the label of the image after the first random transformation.

9. The data enhancement method applied to the field of coating defect detection according to claim 1, characterized in that: In step (7.3), if the ratio of the intersection area to the box area or the ratio of the intersection area to the area of ​​the corresponding labeled box of the image after the first random transformation is still greater than the initial set threshold conf when the process is repeated more than 1000 times, it is judged that the feature image is not suitable for the image after the first random transformation, the feature image is discarded, and steps (7.2) and (7.3) are repeated.

10. The data enhancement method applied to the field of coating defect detection according to claim 1, characterized in that: In step 7, the third random transformation includes random brightness, random Gaussian blur and random noise operations.

11. The data enhancement method applied to the field of coating defect detection according to claim 1, characterized in that: In step (8), if probabilistic splicing occurs, a blank image is first constructed, the size of which is twice the size of the input image of the deep learning network model. Then, the coordinates of the splicing points are randomly selected in the blank image to divide the blank image into 4 blocks. The size of the image to be spliced ​​is adjusted and randomly mapped to the 4 blank images. Finally, the mapped image is scaled and offset, and the annotation label information is adjusted to obtain the spliced ​​image and the corresponding annotation label.

Images