An image data labeling method

Abstract

The image data labeling method disclosed by the application comprises the following steps: calculating the probability distribution value of each category of image data calculated by an integrated model and the probability distribution value of each category of image data calculated by a training model to obtain a first probability distribution average value; calculating the probability distribution value of each category of image data calculated by the integrated model and the probability distribution value of each category of image data calculated by the training model after data enhancement to obtain a second probability distribution average value; calculating the second probability distribution average value and the first probability distribution average value to obtain a third probability distribution average value; selecting a corresponding class activation map according to the category with the maximum probability in the third probability distribution average value; judging whether the class activation map data matches the distribution of the corresponding category data by using the trained CAM model; if the class activation map data matches the distribution of the corresponding category data, a matching label is given, and the image data is retained; and the retained image data is labeled according to the category with the maximum third probability distribution average value and is returned to the training model for iteration.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an image data annotation method. Background Art

[0002] In the era of artificial intelligence dominated by deep learning, the demand for image data is unprecedented. The performance of artificial intelligence models depends largely on the amount of image data and the quality of annotations, which directly determine the lower limit of model accuracy. Therefore, the acquisition of image data and the optimization of annotation methods are particularly important. Figure 1 As shown, the traditional image data annotation method mainly uses a video recording device to collect image data, pre-processes the collected image data, and assigns annotation personnel to manually annotate the data of each image. The data of each image is given a unique label (category), and the annotated image data is input into the training model. The training model is trained, and then the trained training model is evaluated. If the classification accuracy of the test image data input into the training model is greater than the set threshold, the test passes. If not, the image data that is not classified according to the label is collected and data is screened. The screened image data is input into the training model, and the training model annotates the image data. After annotation, it is manually corrected, and the training model is trained again after correction. However, manual correction is still required after annotation, and the labor cost is high, which affects the speed of training model training and image data iteration. Summary of the Invention

[0003] The object of the present invention is to provide an image data annotation method that can reduce labor costs and improve the accuracy of image data correction.

[0004] The image data annotation method of the present invention comprises the following steps:

[0005] S1. Input the labeled image data into the training model for training, and calculate the probability distribution value of each category of the image data through the trained training model;

[0006] S2. Use the test image data to calculate the probability distribution value of each category of the image data and test and evaluate the training model;

[0007] S3. If the test evaluation fails, new image data is collected and data screening is performed on the newly collected image data;

[0008] S4, the filtered data is returned to the training model for annotation, and the probability distribution value of each category of the image data is calculated through the trained training model;

[0009] S5: Input the image data with the categories marked in S1 into multiple ensemble models with different structures and parameters from the training models, train the ensemble models, and use the image data filtered by S3 to return the trained ensemble models to calculate the probability distribution value of each category of the image data.

[0010] S6: Input the labeled image data in S1 into the training model, output the class activation map corresponding to each category of the image data at the designated layer in the training model, and input the class activation map corresponding to each category into the CAM model to train the CAM model;

[0011] S7, ensemble model correction: calculating the probability distribution value of each category of the image data calculated by the ensemble model in S5 and the probability distribution value of each category of the image data calculated by the training model in S4 to obtain a first probability distribution average value;

[0012] S8, consistency regularization correction: perform data augmentation processing on the newly collected image data, and calculate the probability distribution value of each category of the image data through the trained training model. At the same time, the probability distribution value of each category of the image data after the data augmentation is also calculated through the trained ensemble model;

[0013] S9, consistency regularization correction: Calculate the probability distribution value of each category of the image data calculated by the integrated model in S8 and the probability distribution value of each category of the image data calculated by the training model to obtain a second probability distribution average value, and calculate the second probability distribution average value and the first probability distribution average value to obtain a third probability distribution average value;

[0014] S10, CAM model correction: Use the image data filtered by S3 to output the class activation map of each category at the specified layer of the training model, and select the corresponding class activation map according to the category with the largest probability in the average value of the third probability distribution, and pass it to the trained CAM model. The trained CAM model determines whether the class activation map data matches the distribution of the corresponding category data. If it matches, it is assigned a matching label, and if it does not match, it is assigned an unmatched label. The image data with the matching label is retained, otherwise it is discarded;

[0015] S11. Label the retained image data according to the category with the largest mean value of the third probability distribution, pass it to the training model and return to S1 for iteration.

[0016] The image data labeling method provided in the application can reduce the artificial cost and improve the image data correction accuracy by calculating the first probability distribution average value through the integrated model, correcting the image data labeled by the training model, and calculating the second probability distribution average value through the integrated model and the training model, calculating the third probability distribution average value through the second probability distribution average value and the first probability distribution average value, and performing consistency regularization correction, thereby reducing the uncertainty of the model output and further improving the image data correction accuracy. In addition, the class activation map of each class is output by the training model at the specified layer using the image data screened in S3, and the corresponding class activation map is selected according to the class with the maximum probability in the third probability distribution average value, and is transmitted to the trained CAM model. The trained CAM model is used to determine whether the class activation map data matches the distribution of the corresponding class data, and if the match is matched, the matched label is assigned, and if the match is not matched, the unmatched label is assigned. The image data with the matched label is retained, which can effectively improve the accuracy of the image data matching the corresponding class data and further improve the image data correction accuracy. Furthermore, the image data labeled by the training model is corrected through the integrated model, consistency regularization and CAM model, which greatly improves the accuracy of the image data correction, thereby reducing the required artificial cost and improving the speed of model / data iteration.

[0017] As a preferred scheme of the application, the data screening of step S3 is:

[0018] The neuron parameters of the newly collected image data in the trained training model are calculated, and the original score of each image data output by the training model is obtained.

[0019] The original score of each image data output is normalized by using the softmax function to convert the probability distribution value of each class of image data.

[0020] The probability margin difference value of the two classes with the maximum prediction probability of each image data is calculated.

[0021] The probability entropy value of each image data is calculated using the obtained probability distribution value.

[0022] The margin threshold and the entropy threshold are set, the image data with the probability margin difference value greater than the margin threshold and the probability entropy value less than the entropy threshold is filtered, and the last remaining image data is screened out.

[0023] As a preferred solution of the present invention, the specific calculation formula for the average value of the first probability distribution is: Among them, p o is the probability distribution value of each category of image data calculated by the training model in S4, p i The probability distribution value of each category of image data calculated by the ith integrated model of S5, w i is its corresponding weighting coefficient.

[0024] As a preferred solution of the present invention, the specific calculation formula for the average value of the second probability distribution is: Among them, p g To train the model, calculate the probability distribution value of each category of the image data in S8, p j The probability distribution value of each category of the image data calculated by the j-th integrated model S8.

[0025] As a preferred solution of the present invention, the data enhancement processing of the newly collected image data specifically includes but is not limited to the following:

[0026] Perform image flipping on the newly collected image data;

[0027] Crop the flipped image data;

[0028] Perform color dithering on the cropped image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flowchart of the traditional image data annotation method;

[0030] Figure 2 The figure is a flow chart of an image data annotation method according to the present invention. DETAILED DESCRIPTION

[0031] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0032] A method for labeling image data, such as Figure 2 As shown, it includes S1, inputting the image data with the marked categories into the training model for training, and calculating the probability distribution value of each category of the image data through the trained training model.

[0033] Image data can be acquired through a camera or other device and stored in a computer. In the early stages, each image data is manually labeled and assigned a unique label, such as label a for image data of category 1 and label b for image data of category 2. The labeled image data is then fed into a training model for training. The trained model calculates the probability distribution value for each category of the image data.

[0034] S2. Use the test image data to calculate the probability distribution value of each category of the image data and test and evaluate the training model.

[0035] The test image data is input into the training model and the probability distribution value of each category of the image data is calculated. If the classification accuracy of the test image data is greater than the set threshold, the training model evaluation passes, otherwise the evaluation fails.

[0036] S3. If the test evaluation fails, new image data is collected and data screening is performed on the newly collected image data.

[0037] The data screening is as follows: the neuron parameters of the newly collected image data in the trained model are calculated, and the raw score of each image data output by the training model is obtained; then the raw score of each image data output is normalized using the softmax function and converted into the probability distribution value of each category of the image data. The formula is: where z i is the original score of the i-th category output. Calculate the probability marginal difference between the two categories with the largest predicted probability for each image data. The calculation formula is: Probability marginal difference = p1-p2. Using the obtained probability distribution value, calculate the probability entropy value of each image data. The calculation formula is: By setting a margin threshold and an entropy threshold, we filter out image data with a probability margin difference greater than the margin threshold and a probability entropy value less than the entropy threshold, thus selecting the remaining image data. This method can effectively improve the effectiveness and speed of image data screening. In addition, it not only helps reduce the labeling workload but also maximizes the learning efficiency of the training model.

[0038] S4. The filtered data is returned to the training model for labeling. The trained model is used to calculate the probability distribution value for each category of the image data. The training model specifically labels, for example, if the probability of category a in image data 1 is the highest, it is assigned the corresponding label a, and if the probability of category b in image data 2 is the highest, it is assigned the corresponding label b. The trained training model calculates the probability distribution value for each category of the image data. At this point, it is not known whether the labeled label a or b matches the category of image data 1 or the category of image data 2, so subsequent corrections are required to improve the accuracy of the training model labeling.

[0039] S5: Input the image data with categories marked by S1 into multiple integrated models with different structures and parameters from the training models, train the integrated models, and use the image data filtered by S3 to return the trained integrated models to calculate the probability distribution value of each category of the image data.

[0040] To effectively calibrate the trained model's image data annotations, an ensemble model is introduced. This ensemble model consists of multiple models with different structures and parameters than the trained model. Before calibration, the ensemble model must be trained using the same training data as the trained model, specifically the image data labeled by S1. Once the ensemble model is trained, the image data filtered by S3 is fed back into the trained ensemble model to calculate the probability distribution for each category in the image data.

[0041] S6. Input the labeled image data in S1 into the training model, output the class activation map corresponding to each category of the image data at the specified layer in the training model, and input the class activation map corresponding to each category into the CAM model (ClassActivation Mapping) model to train the CAM model.

[0042] In addition, in order to further improve the accuracy of correction, the CAM model is introduced. The training model outputs the class activation map corresponding to each category of the image data in the specified layer of the training model for the image data with labels S1, and then inputs the class activation map corresponding to each category into the CAM model to train the CAM model, as shown in the formula Among them, CAM i is the class activation map of the i-th category, It is a feature map k If the model specifies the layer as the last output layer of the model, the class activation map and the training model output original score z i The relationship is as follows: i =∑ x, y CAM i (x, y), by training model weights or gradients, etc., the influence of each feature map is weighted and summed. The class activation map can effectively project high-dimensional features into a spatial map that only highlights the most relevant areas. The class activation map corresponding to the spatial map of each relevant area can identify the label corresponding to each image data category. It has strong robustness due to the dimensionality reduction processing of image data.

[0043] S7, integrated model correction: the probability distribution value of each category of the image data calculated by the integrated model in S5 and the probability distribution value of each category of the image data calculated by the training model in S4 are calculated to obtain a first probability distribution average value.

[0044] The original score of each image data is obtained by calculating the neuron parameters of the filtered data in the training model, and the raw score of each image data output is normalized using the softmax function to convert it into a probability distribution value for each category of the image data. The original score of each image data is obtained by calculating the neuron parameters of the filtered image data in the integrated model, and the raw score of each image data output is normalized using the softmax function to convert it into a probability distribution value for each category of the image data. Then, the probability distribution value of each category of the image data calculated by the integrated model is calculated with the probability distribution value of each category of the image data calculated by the training model to obtain the first probability distribution average value, which is calculated as follows: Among them, p o is the probability distribution value of each category of image data calculated by the training model in S4, p i The probability distribution value of each category of image data calculated by the ith integrated model of S5, w i The above method can further improve the accuracy of image data correction, thereby reducing labor costs.

[0045] S8. Consistency regularization correction: Perform data augmentation on the newly collected image data, and use the trained model to calculate the probability distribution value of each category of the image data after the data augmentation. At the same time, the probability distribution value of each category of the image data after the data augmentation is also calculated through the trained integration model.

[0046] The data enhancement processing mainly involves flipping the newly collected image data, then cropping the flipped image data, and performing color jittering on the cropped image data.

[0047] Data augmentation is performed on the newly collected image data. The raw scores of each image data are obtained by calculating the neuron parameters in the trained model for the data augmented image data. The raw scores of each image data are normalized using the softmax function to convert them into probability distribution values ​​for each category of the image data. In addition, data augmentation is performed on the newly collected image data. The raw scores of each image data are obtained by calculating the neuron parameters in the trained integrated model for the data augmented image data. The raw scores of each image data are normalized using the softmax function to convert them into probability distribution values ​​for each category of the image data.

[0048] S9, consistency regularization correction: calculate the probability distribution value of each class of image data calculated by the integrated model in S8 with the probability distribution value of each class of image data calculated by the training model to obtain a second probability distribution average value, and calculate the second probability distribution average value with the first probability distribution average value to obtain a third probability distribution average value.

[0049] The specific calculation formula of the second probability distribution average value is: Where, p g is the probability distribution value of each class of image data calculated by the training model in S8, p j is the probability distribution value of each class of image data calculated by the jth integrated model in S8. The above method can further improve the accuracy of image data correction, thereby reducing the labor cost; in addition, the introduction of data enhancement increases the noise and diversity of the image, thereby improving the robustness of the model.

[0050] S10, CAM model correction: using the image data selected in S3 to output the class activation map of each class at the specified layer of the training model, and selecting the corresponding class activation map according to the class with the maximum probability in the third probability distribution average value, and passing it to the trained CAM model. The trained CAM model judges whether the class activation map data matches the distribution of the corresponding class data, and if it matches, it gives a matching label, and if it does not match, it gives a non-matching label. The image data with matching label is retained, otherwise it is discarded.

[0051] S11, according to the class with the maximum third probability distribution average value, the retained image data is given a label. And pass it to the training model to return to S1 for iteration.

[0052] The correction accuracy table is as follows:

[0053] Integrated model calibration CAM model correction Consistency Regularization Correction Accuracy (%) 38 √ 48 √ √ 87 √ √ √ 89

[0054] From the correction accuracy table, it can be seen that the accuracy rate can reach 48% by adding the integrated model correction, the accuracy rate can reach 87% by adding the CAM model correction, and the accuracy rate can reach 89% by adding the consistency regularization correction, thereby greatly improving the accuracy of image data label correction. In addition, due to the use of active learning, the amount of data annotation required on the self-built data set is less than 10% of the original data set, thereby improving the speed of image data correction. In addition, the use of automatic annotation and automatic class correction greatly reduces the required labor cost and improves the speed of model / data iteration.

[0055] The above embodiments are intended only to illustrate the detailed embodiments of the present invention. The present invention is not limited to the detailed embodiments described above, nor does it necessarily rely on the detailed embodiments to be implemented. Those skilled in the art will appreciate that any improvements to the present invention, equivalent replacements for the raw materials of the present invention, additions of auxiliary ingredients, and selection of specific methods are all within the scope of protection and disclosure of the present invention.

Claims

1. A method for labeling image data, comprising the following steps: S1. Input the labeled image data into the training model for training, and calculate the probability distribution value of each category of the image data through the trained training model; S2. Use the test image data to calculate the probability distribution value of each category of the image data and test and evaluate the training model; S3. If the test evaluation fails, new image data is collected and data screening is performed on the newly collected image data; S4, the filtered data is returned to the training model for annotation, and the probability distribution value of each category of the image data is calculated through the trained training model; S5: Input the image data with the categories marked in S1 into multiple ensemble models with different structures and parameters from the training models, train the ensemble models, and use the image data filtered by S3 to return the trained ensemble models to calculate the probability distribution value of each category of the image data. S6: Input the image data marked with categories in S1 into the training model, output the class activation map corresponding to each category of the image data at the designated layer in the training model, and input the class activation map corresponding to each category into the CAM model to train the CAM model; S7, ensemble model correction: calculating the probability distribution value of each category of the image data calculated by the ensemble model in S5 and the probability distribution value of each category of the image data calculated by the training model in S4 to obtain a first probability distribution average value; S8, consistency regularization correction: perform data augmentation processing on the newly collected image data, and calculate the probability distribution value of each category of the image data through the trained training model. At the same time, the probability distribution value of each category of the image data after the data augmentation is also calculated through the trained ensemble model; S9, consistency regularization correction: Calculate the probability distribution value of each category of the image data calculated by the integrated model in S8 and the probability distribution value of each category of the image data calculated by the training model to obtain a second probability distribution average value, and calculate the second probability distribution average value and the first probability distribution average value to obtain a third probability distribution average value; S10, CAM model correction: Use the image data filtered by S3 to output the class activation map of each category at the specified layer of the training model, and select the corresponding class activation map according to the category with the largest probability in the average value of the third probability distribution, and pass it to the trained CAM model. The trained CAM model determines whether the class activation map data matches the distribution of the corresponding category data. If it matches, it is assigned a matching label, and if it does not match, it is assigned a non-matching label. The image data with the matching label is retained, otherwise it is discarded; S11. Label the retained image data according to the category with the largest mean value of the third probability distribution, pass it to the training model and return to S1 for iteration.

2. The image data annotation method according to claim 1, characterized in that: The data screening of step S3 is as follows: Calculate the neuron parameters in the trained model for the newly collected image data and obtain the original score of each image data output by the training model; Use the softmax function to normalize the original score of each output image data and convert it into the probability distribution value of each category of the image data; Calculate the probability marginal difference between the two categories with the largest predicted probability for each image data; Using the obtained probability distribution value, calculate the probability entropy value of each image data; Set the marginal threshold and entropy threshold, filter out the image data whose probability marginal difference value is greater than the marginal threshold and whose probability entropy value is less than the entropy threshold, and screen out the remaining image data.

3. The image data annotation method according to claim 1, characterized in that: The specific calculation formula for the average value of the first probability distribution is: ,in, The probability distribution value of each category of the image data calculated by the training model in S4, For S5 The probability distribution value of each category of image data calculated by the integrated model, is its corresponding weighting coefficient.

4. The image data annotation method according to claim 1, characterized in that: The specific calculation formula for the mean value of the second probability distribution is: ,in, To train the model, calculate the probability distribution value of each category of the image data in S8. For S8 The probability distribution value of each category of the image data calculated by the integrated model.

5. The image data annotation method according to claim 1, characterized in that: The data enhancement processing of the newly collected image data specifically includes: the following: Perform image flipping on the newly collected image data; Crop the flipped image data; Perform color dithering on the cropped image data.

Images