Image data annotation method

Abstract

The invention discloses an image data labeling method. The method comprises the following steps: calculating a probability distribution value of each category of image data calculated by an integrated model and a probability distribution value of each category of image data calculated by a training model to obtain a first probability distribution average value; calculating the probability distribution value of each category of the image data calculated in the integration model and the probability distribution value of each category of the image data calculated in the training model to obtain a second probability distribution average value, and calculating the first probability distribution average value to obtain a third probability distribution average value; selecting a corresponding class activation graph according to the class with the maximum probability in the third probability distribution average value, judging whether the class activation graph data is matched with the distribution of the corresponding class data or not through a trained CAM model, giving a matching mark if matching is carried out, and retaining the image data. And according to the category with the maximum third probability distribution average value, endowing a label to the reserved image data and returning the training model for iteration.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an image data labeling method. Background Art

[0002] In the era of artificial intelligence dominated by deep learning, the demand for image data is unprecedentedly huge. The performance of artificial intelligence models depends largely on the amount of image data and the quality of annotation, which directly determine the lower limit of model accuracy. Therefore, the acquisition of image data and the optimization of annotation methods are particularly important. Figure 1 As shown, the traditional image data annotation method mainly uses a video recording device to collect image data, pre-processes the collected image data, and assigns annotators to manually annotate the data of each image. Each image data is given a unique label (category), and the annotated image data is input into the training model, the training model is trained, and then the trained training model is evaluated. If the classification accuracy of the test image data input into the training model is greater than the set threshold, the test passes. If not, the image data that is not classified according to the label is collected and data is screened, and the screened image data is input into the training model. The training model annotates the image data, and then manually corrects it after annotation, and then trains the training model after correction. However, manual correction is still required after annotation, and the labor cost is high, which affects the speed of training model training and image data iteration. Summary of the invention

[0003] The object of the present invention is to provide an image data annotation method which can reduce labor costs and improve the accuracy of image data correction.

[0004] The image data annotation method of the present invention comprises the following steps:

[0005] S1. Input the image data with labeled categories into the training model for training, and calculate the probability distribution value of each category of the image data through the trained training model;

[0006] S2. Use the test image data to calculate the probability distribution value of each category of the image data and test and evaluate the training model;

[0007] S3. If the test evaluation fails, new image data is collected and data screening is performed on the newly collected image data;

[0008] S4, returning the filtered data to the training model for annotation, and calculating the probability distribution value of each category of the image data through the trained training model;

[0009] S5, input the image data with the categories marked by S1 into multiple integrated models with different structures and parameters from the training models, train the integrated models, use the image data filtered by S3 to return the trained integrated models, and calculate the probability distribution value of each category of the image data;

[0010] S6, input the image data marked with labels in S1 into the training model, output the class activation map corresponding to each category of the image data at the specified layer in the training model, and input the class activation map corresponding to each category into the CAM model to train the CAM model;

[0011] S7, integrated model correction: the probability distribution value of each category of the image data calculated by the integrated model in S5 and the probability distribution value of each category of the image data calculated by the training model in S4 are calculated to obtain a first probability distribution average value;

[0012] S8, consistency regularization correction: perform data enhancement processing on the newly collected image data, and calculate the probability distribution value of each category of the image data through the trained training model for the image data after data enhancement. At the same time, the probability distribution value of each category of the image data after data enhancement is also calculated through the trained integrated model;

[0013] S9, consistency regularization correction: the probability distribution value of each category of the image data calculated by the integrated model in S8 and the probability distribution value of each category of the image data calculated by the training model are calculated to obtain a second probability distribution average value, and the second probability distribution average value and the first probability distribution average value are calculated to obtain a third probability distribution average value;

[0014] S10, CAM model correction: Use the image data filtered by S3 to output the class activation map of each category in the specified layer of the training model, and select the corresponding class activation map according to the category with the largest probability in the average value of the third probability distribution, and pass it to the trained CAM model. The trained CAM model determines whether the class activation map data matches the distribution of the corresponding category data. If it matches, it is assigned a matching label, and if it does not match, it is assigned a non-matching label. The image data with the matching label is retained, otherwise it is discarded;

[0015] S11. Assign labels to the retained image data according to the category with the largest mean value of the third probability distribution, pass it to the training model and return to S1 for iteration.

[0016] The image data labeling method of the present invention uses an integrated model to calculate the probability distribution value of each category of the image data calculated by the integrated model in S5 and the probability distribution value of each category of the image data calculated by the training model in S4 to obtain a first probability distribution average value to correct the image data of the labeled category of the training model, thereby reducing labor costs and improving the accuracy of image data correction; in addition, by calculating the probability distribution value of each category of the image data calculated by the integrated model in S8 and the probability distribution value of each category of the image data calculated by the training model to obtain a second probability distribution average value, the second probability distribution average value is calculated with the first probability distribution average value to obtain a third probability distribution average value for consistency regularization correction, thereby reducing the uncertainty of the model output and further improving the accuracy of image data correction; In addition, the image data filtered by S3 is used to output the class activation map of each category in the specified layer of the training model, and the corresponding class activation map is selected according to the category with the largest probability in the average value of the third probability distribution, and passed to the trained CAM model. The trained CAM model determines whether the class activation map data matches the distribution of the corresponding category data. If it matches, a matching label is assigned, and if it does not match, a non-matching label is assigned. The image data with the matching label is retained, which can effectively improve the accuracy of the image data matching the corresponding category data, and further improve the accuracy of image data correction. Furthermore, the image data of the labeled category of the training model is corrected by integrating the model, consistency regularization and CAM model, which greatly improves the accuracy of image data correction, thereby reducing the required labor cost and improving the speed of model / data iteration.

[0017] As a preferred solution of the present invention, the data screening in step S3 is as follows:

[0018] Calculate the neuron parameters in the trained model for the newly collected image data, and obtain the original score of each image data output by the training model;

[0019] The softmax function is used to normalize the original scores of each output image data and convert them into probability distribution values ​​of each category of the image data;

[0020] Calculate the probability marginal difference between the two categories with the largest predicted probability for each image data;

[0021] Using the obtained probability distribution value, calculate the probability entropy value of each image data;

[0022] Set the marginal threshold and entropy threshold, filter out the image data whose probability marginal difference value is greater than the marginal threshold and whose probability entropy value is less than the entropy threshold, and select the remaining image data.

[0023] As a preferred solution of the present invention, the specific calculation formula for the average value of the first probability distribution is: Among them, p o is the probability distribution value of each category of the image data calculated by the training model in S4, p i is the probability distribution value of each category of image data calculated by the i-th integrated model of S5, w i is its corresponding weighting coefficient.

[0024] As a preferred solution of the present invention, the specific calculation formula for the average value of the second probability distribution is: Among them, p g To train the model, calculate the probability distribution value of each category of the image data in S8, p j The probability distribution value of each category of the image data calculated by the j-th integrated model S8.

[0025] As a preferred solution of the present invention, the data enhancement processing of the newly collected image data specifically includes but is not limited to the following:

[0026] Perform image flipping on the newly collected image data;

[0027] Crop the flipped image data;

[0028] Perform color dithering on the cropped image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of the traditional image data annotation method;

[0030] Figure 2 The figure is a flow chart of an image data annotation method according to the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] A method for labeling image data, such as Figure 2 As shown, it includes S1, inputting the image data with labeled categories into the training model for training, and calculating the probability distribution value of each category of the image data through the trained training model.

[0033] Image data can be obtained through a camera device, etc., and the image data can be put into a computer, etc. In the early stage, the category of each image data is manually labeled and assigned a unique label, such as image data category 1 is given a corresponding label a, and image data category 2 is given a corresponding label b. Then the labeled image data is input into the training model for training, and the probability distribution value of each category of the image data is calculated through the trained training model.

[0034] S2. Use the test image data to calculate the probability distribution value of each category of the image data and test and evaluate the training model.

[0035] The test image data is input into the training model to calculate the probability distribution value of each category of the image data. If the classification accuracy of the test image data is greater than the set threshold, the training model evaluation passes, otherwise the evaluation fails.

[0036] S3. If the test evaluation fails, new image data is collected and data screening is performed on the newly collected image data.

[0037] The specific data screening is as follows: the neuron parameters of the newly collected image data in the trained model are calculated, and the original score of each image data output by the training model is obtained; then the original score of each image data output is normalized using the softmax function and converted into the probability distribution value of each category of the image data. The formula is: where z i is the original score of the output of the i-th category. Calculate the probability marginal difference of the two categories with the largest prediction probability for each image data, and the calculation formula is: probability marginal difference = p1-p2. Using the obtained probability distribution value, calculate the probability entropy value of each image data, and the calculation formula is: Set the marginal threshold and entropy threshold, filter the image data with probability marginal difference greater than the marginal threshold and probability entropy less than the entropy threshold, and filter out the remaining image data. The above method can effectively improve the effectiveness and speed of image data screening. In addition, it not only helps to reduce the workload of annotation, but also maximizes the learning efficiency of the training model.

[0038] S4. The filtered data is returned to the training model for annotation, and the probability distribution value of each category of the image data is calculated through the trained training model. The specific annotation of the training model is that if the probability of image data 1 category a is the largest, the corresponding label a is given, and if the probability of image data 2 category b is the largest, the corresponding label b is given. The probability distribution value of each category of the image data is calculated through the trained training model. At this time, it is not known whether the annotated label a or b matches the image data category 1 or the image data category 2, so correction is required later to improve the accuracy of the training model annotation.

[0039] S5: Input the image data with categories marked by S1 into multiple integrated models with different structures and parameters from the training models, train the integrated models, use the image data filtered by S3 to return the trained integrated models, and calculate the probability distribution value of each category of the image data.

[0040] In order to effectively correct the annotation of the training model on the image data, an integrated model is introduced. The integrated model is a model with multiple structures and parameters different from the training model. The integrated model needs to be trained before correction. The training process uses the same training data as the training model, that is, the image data with the categories annotated by S1 as the training data. After the integrated model is trained, the image data filtered by S3 is used to return the trained integrated model to calculate the probability distribution value of each category of the image data.

[0041] S6. Input the image data marked with labels in S1 into the training model, output the class activation map corresponding to each category of the image data at the specified layer in the training model, and input the class activation map corresponding to each category into the CAM model (ClassActivation Mapping) model to train the CAM model.

[0042] In addition, in order to further improve the accuracy of correction, the CAM model is introduced. The training model outputs the class activation map corresponding to each category of the image data in the specified layer of the training model for the image data labeled by S1, and then inputs the class activation map corresponding to each category into the CAM model to train the CAM model, as shown in the formula Among them, CAM i is the class activation map of the ith category, It is a feature map k If the model specifies the layer as the final output layer of the model, the class activation map and the original score z output by the training model i The relationship is as follows: i =∑ x, y CAM i (x, y), by training model weights or gradients, etc., the influence of each feature map is weighted summed. The class activation map can effectively project high-dimensional features into a spatial map that only highlights the most relevant areas. The class activation map corresponding to the spatial map of each relevant area can identify the label corresponding to each image data category. It has strong robustness due to the dimensionality reduction processing of image data.

[0043] S7, integrated model correction: the probability distribution value of each category of the image data calculated by the integrated model in S5 and the probability distribution value of each category of the image data calculated by the training model in S4 are calculated to obtain a first probability distribution average value.

[0044] The original score of each image data is obtained by calculating the neuron parameters of the filtered data in the training model, and the original score of each image data output is normalized using the softmax function to convert it into a probability distribution value for each category of the image data. The original score of each image data is obtained by calculating the neuron parameters of the filtered image data in the integrated model, and the original score of each image data output is normalized using the softmax function to convert it into a probability distribution value for each category of the image data. Then, the probability distribution value of each category of the image data calculated by the integrated model and the probability distribution value of each category of the image data calculated by the training model are calculated to obtain the first probability distribution average value, and the calculation formula is: Among them, p o is the probability distribution value of each category of the image data calculated by the training model in S4, p i is the probability distribution value of each category of image data calculated by the i-th integrated model of S5, w i The above method can further improve the accuracy of image data correction, thereby reducing labor costs.

[0045] S8. Consistency regularization correction: Perform data enhancement processing on the newly collected image data, and calculate the probability distribution value of each category of the image data through the trained training model. At the same time, the probability distribution value of each category of the image data after data enhancement is also calculated through the trained integration model.

[0046] The data enhancement processing mainly involves flipping the newly collected image data, then cropping the flipped image data, and performing color jittering on the cropped image data.

[0047] The newly collected image data is subjected to data enhancement processing, and the original score of each image data is obtained by calculating the neuron parameters of the data-enhanced image data in the training model, and the original score of each image data output is normalized using the softmax function to convert it into the probability distribution value of each category of the image data. In addition, the newly collected image data is subjected to data enhancement processing, and the original score of each image data is obtained by calculating the neuron parameters of the data-enhanced image data in the trained integrated model, and the original score of each image data output is normalized using the softmax function to convert it into the probability distribution value of each category of the image data.

[0048] S9, consistency regularization correction: calculate the probability distribution value of each category of the image data calculated by the integrated model in S8 and the probability distribution value of each category of the image data calculated by the training model to obtain a second probability distribution average, and calculate the second probability distribution average and the first probability distribution average to obtain a third probability distribution average.

[0049] The specific calculation formula for the average value of the second probability distribution is: Among them, p g To train the model, calculate the probability distribution value of each category of the image data in S8, p j The probability distribution value of each category of image data calculated by the jth integrated model of S8. The above method can further improve the accuracy of image data correction, thereby reducing labor costs; in addition, the introduction of data enhancement increases the noise and diversity of the image, thereby improving the robustness of the model.

[0050] S10, CAM model correction: Use the image data filtered by S3 to output the class activation map of each category in the specified layer of the training model, and select the corresponding class activation map according to the category with the largest probability in the average value of the third probability distribution, and pass it to the trained CAM model. The trained CAM model determines whether the class activation map data matches the distribution of the corresponding category data. If it matches, it is assigned a matching label, and if it does not match, it is assigned a non-matching label. The image data with the matching label is retained, otherwise it is discarded.

[0051] S11, assign labels to the retained image data according to the category with the largest mean value of the third probability distribution, and pass it to the training model and return to S1 for iteration.

[0052] The calibration accuracy table is as follows:

[0053] Integrated model calibration CAM model correction Consistency Regularization Correction Accuracy (%) 38 √ 48 √ √ 87 √ √ √ 89

[0054] From the correction accuracy table, we can see that the accuracy can reach 48% when the integrated model correction is added, and the accuracy can reach 87% when the CAM model correction is added, and the accuracy can reach 89% when the consistency regularization correction is added, which greatly improves the accuracy of image data label correction. In addition, due to the use of active learning, the amount of data annotation required on the self-built data set is less than 10% of the original data set, thereby improving the speed of image data correction. In addition, the use of automatic annotation and automatic category correction has greatly reduced the labor cost required and improved the speed of model / data iteration.

[0055] The above embodiments are only used to illustrate the detailed scheme of the present invention, and the present invention is not limited to the above detailed scheme, that is, it does not mean that the present invention must rely on the above detailed scheme to be implemented. Those skilled in the art should understand that any improvement of the present invention, equivalent replacement of the raw materials of the product of the present invention, addition of auxiliary components, selection of specific methods, etc., are all within the protection scope and disclosure scope of the present invention.

Claims

1. A method for labeling image data, comprising the following steps: S1. Input the image data with labeled categories into the training model for training, and calculate the probability distribution value of each category of the image data through the trained training model; S2. Use the test image data to calculate the probability distribution value of each category of the image data. Test and evaluate the training model; S3. If the test evaluation fails, new image data is collected and data screening is performed on the newly collected image data; S4, returning the filtered data to the training model for annotation, and calculating the probability distribution value of each category of the image data through the trained training model; S5, input the image data with the categories marked by S1 into multiple integrated models with different structures and parameters from the training models, train the integrated models, use the image data filtered by S3 to return the trained integrated models, and calculate the probability distribution value of each category of the image data; S6, input the image data with the categories marked in S1 into the training model, output the class activation map corresponding to each category of the image data at the specified layer in the training model, and input the class activation map corresponding to each category into the CAM model to train the CAM model; S7, integrated model correction: the probability distribution value of each category of the image data calculated by the integrated model in S5 and the probability distribution value of each category of the image data calculated by the training model in S4 are calculated to obtain a first probability distribution average value; S8, consistency regularization correction: perform data enhancement processing on the newly collected image data, and calculate the probability distribution value of each category of the image data through the trained training model for the image data after data enhancement. At the same time, the probability distribution value of each category of the image data after data enhancement is also calculated through the trained integrated model; S9, consistency regularization correction: the probability distribution value of each category of the image data calculated by the integrated model in S8 and the probability distribution value of each category of the image data calculated by the training model are calculated to obtain a second probability distribution average value, and the second probability distribution average value and the first probability distribution average value are calculated to obtain a third probability distribution average value; S10, CAM model correction: Use the image data filtered by S3 to output the class activation map of each category at the specified layer of the training model, and select the corresponding class activation map according to the category with the largest probability in the average value of the third probability distribution, and pass it to the trained CAM model. The trained CAM model determines whether the class activation map data matches the distribution of the corresponding category data. If it matches, a matching tag is assigned, and if it does not match, a non-matching tag is assigned. The image data of the matching tag is retained, otherwise it is discarded; S11. Assign labels to the retained image data according to the category with the largest mean value of the third probability distribution, pass it to the training model and return to S1 for iteration.

2. The image data annotation method according to claim 1, characterized in that: The data screening of step S3 is as follows: Calculate the neuron parameters in the trained model for the newly collected image data, and obtain the original score of each image data output by the training model; The softmax function is used to normalize the original scores of each output image data and convert them into probability distribution values ​​of each category of the image data; Calculate the probability marginal difference between the two categories with the largest predicted probability for each image data; The obtained probability distribution value is used to calculate the probability entropy value of each image data; the marginal threshold and entropy threshold are set to filter out the image data with a probability marginal difference greater than the marginal threshold and a probability entropy value less than the entropy threshold, and finally filter out the remaining image data.

3. The image data annotation method according to claim 1, characterized in that: The specific calculation formula of the average value of the first probability distribution is: Among them, p o is the probability distribution value of each category of the image data calculated by the training model in S4, p i is the probability distribution value of each category of image data calculated by the i-th integrated model of S5, w i is its corresponding weighting coefficient.

4. The image data annotation method according to claim 1, characterized in that: The specific calculation formula for the average value of the second probability distribution is: Among them, p g To train the model, calculate the probability distribution value of each category of the image data in S8, p j The probability distribution value of each category of the image data calculated by the j-th integrated model S8.

5. The image data annotation method according to claim 1, characterized in that: The data enhancement processing of the newly collected image data specifically includes but is not limited to the following: Perform image flipping on the newly collected image data; Crop the flipped image data; Perform color dithering on the cropped image data.

Images