Image annotation method and system
By using image annotation method in the object detection task, and using object detection model and confidence learning to perform automated data annotation inspection and correction, the problem of object detection data annotation quality is solved, and the model accuracy and manual efficiency are improved.
Patent Information
- Application Number
- CN202311253239.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-26
- Publication Date
- 2025-04-04
AI Technical Summary
The existing technology lacks effective automated data annotation inspection methods in target detection tasks, resulting in serious data annotation quality problems and affecting model optimization tasks.
An image annotation method is proposed, using the object detection model to train and forward prediction of image data, combine confidence learning to calculate the evaluation score, automatically identify and correct labeling errors, and improve data quality and model accuracy through iterative training and error correction penalization loss function.
This method can quickly reduce the missed marking and mislabeling problems in image annotation, significantly improve the accuracy of the target detection model, reduce labor costs and improve efficiency.
Smart Images
Figure CN119763114A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image data processing, and in particular relates to an image annotation method and system. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] With the vigorous development of deep learning, more and more surveillance data is being acquired. Manual labeling of surveillance video data can provide important data support for subsequent intelligent recognition tasks. However, due to manual negligence or different understandings and perceptions of data labelers (for example, for the same target, A thinks it is the target, but B thinks it is not the target), there are often certain quality problems in data labeling. As the amount of data continues to increase, more and more problematic data will accumulate, and data quality issues will become extremely serious. Once there is a problem with the data label, it will also bring certain difficulties to the later model optimization tasks.
[0004] At present, when there are problems with data labeling, there are two ways to proceed. The first method is to manually check and verify the labels one by one. However, since manual inspection will inevitably result in errors or cognitive differences, sometimes data labeling even requires three or more data verifications. Generally, the number of images a person can view in a day is very limited, and manual inspection and verification of data labeling becomes very inefficient. Therefore, data labeling under large data volumes will inevitably lead to greater manpower loss and labor costs. The second method is to automatically check data labels. The introduction of confidence learning has opened a precedent for automated inspection of data labels, allowing data inspection to be further automated. By constructing a probabilistic joint distribution matrix of annotations and predictions, incorrect labels are found and cleaned up. The first method is slow but accurate, and the second method is fast, but currently confidence learning inspection data can generally be implemented in limited tasks, such as classification tasks and regression tasks. There are no relevant cases and implementation plans for target detection tasks. Summary of the invention
[0005] In order to solve the above problems, the present invention proposes an image annotation method and system, which can greatly improve the accuracy of the target detection model and quickly improve the quality of data.
[0006] According to some embodiments, the present invention adopts the following technical solutions:
[0007] An image annotation method comprises the following steps:
[0008] Using the acquired image data and the manually annotated data of the image data, training the target detection model;
[0009] Use the trained target detection model to perform forward prediction on all image data to obtain the corresponding prediction results;
[0010] Calculate evaluation scores for all image data based on confidence learning;
[0011] If the evaluation score of an image is greater than the set threshold, the image is considered to have problems, and all problematic images are recorded and a new list is formed;
[0012] Replacing the label of the image in the new list with the prediction result corresponding to the image;
[0013] Return to the step of training the target detection model and iterate until the number in the currently formed new list meets the set conditions.
[0014] As an optional implementation, before training the target detection model, the following steps are included:
[0015] Set the number of training iterations;
[0016] The training data is augmented by rotating, translating, and / or randomly flipping images within a set range.
[0017] As an alternative implementation, the specific process of calculating the evaluation scores of all image data based on confidence learning includes:
[0018] For each image’s n annotations and m predictions, the IoU (Intersection over Union) values are calculated to obtain the IoU matrix.
[0019] For n annotations and m predictions, distance calculations are performed to obtain distance matrices;
[0020] Based on the IoU matrix and the distance matrix, a similarity matrix is calculated;
[0021] Based on the IoU matrix, establish a question data matrix;
[0022] The data matrix and the similarity matrix are divided according to the question, and the final score is calculated.
[0023] As a further step, the calculation process of the IoU matrix includes:
[0024]
[0025] Among them, the value of the i-th annotation box is represented by D i =(x d1 ,y d1 ,x d2,y d2 ,c d ), where x d1 is the horizontal coordinate of the upper left corner of the target bounding box, y d1 is the upper left corner ordinate of the target bounding box, x d2 is the horizontal coordinate of the lower right corner of the target bounding box, y d2 is the ordinate of the lower right corner of the target bounding box, c d is the label value of the bounding box;
[0026] The value of the i-th prediction box is represented by P i =(x p1 ,y p1 ,x p2 ,y p2 ,c p ,p), where x p1 is the horizontal coordinate of the upper left corner of the target bounding box, y p1 is the upper left corner ordinate of the target bounding box, x p2 is the horizontal coordinate of the lower right corner of the target bounding box, y p2 The ordinate of the lower right corner of the target bounding box, c p is the predicted label value, and p is the predicted probability value.
[0027] As a further step, the calculation process of each element in the distance matrix includes:
[0028]
[0029] where x dc ,y dc ,x pc ,y pc They are respectively the center point position of the i-th annotation box and the center point position of the j-th detection target.
[0030] As a further step, the similarity matrix is:
[0031] M sim =τ×M iou +(1-τ)×(1-M dis )
[0032] Where τ is the relaxation factor, M iou is the IoU matrix, M dis is the distance matrix.
[0033] As a further step, the question score matrix M score for:
[0034] If M IoU The IoU is greater than 0.5, and c d =c p The index of Mscore The position is set to 0;
[0035] If IoU is greater than 0.5, c d ≠c p The index of M score The position is set to 1-p, where p is the predicted probability value.
[0036] If IoU is not greater than 0.5, in M score The position is set to 0 or 1-p, and 1-p is used where there is a prediction result.
[0037] Final score
[0038]
[0039] Where E is represented by an m×n matrix with all 1s.
[0040] As an optional implementation, it also includes:
[0041] Construct an error correction penalty loss function and use it to punish erroneous data training during training;
[0042] The error correction penalty loss function is:
[0043]
[0044] Among them, Ls is the overall loss, Ls D is the loss of the manual annotation part that is considered to have no wrong annotations, Ls I It is the loss that is suspected to be wrong in manual annotation. is the penalty factor, I1 is the problematic list in the data list, and L-I1 refers to the loss of all lists that are considered to have no incorrect annotations.
[0045] As a further step, each loss includes the sum of classification loss, bbox loss and IoU calculation loss.
[0046] As an alternative implementation, the set condition is that the value is less than a set value or no longer decreases.
[0047] An image annotation system, comprising:
[0048] A model training module is configured to train the target detection model using the acquired image data and the manually annotated data of the image data;
[0049] A prediction module is configured to use the trained object detection model to perform forward prediction on all image data to obtain corresponding prediction results;
[0050] An evaluation module is configured to calculate evaluation scores of all image data based on confidence learning; if the evaluation score of an image is greater than a set threshold, the image is considered to have a problem, all images with problems are recorded, and a new list is formed;
[0051] A correction module, configured to replace the label of the image in the new list with the prediction result corresponding to the image;
[0052] The iteration module is configured to call the model training module and iterate until the number of the new list currently formed meets the set conditions.
[0053] Compared with the prior art, the present invention has the following beneficial effects:
[0054] The present invention proposes an automated inspection method for target detection data, adds an error correction penalty loss function and continues iterative training. After multiple data corrections and model iterative training, the purpose of model optimization can be further achieved, the number of problems such as missing and mislabeling in image annotation can be quickly reduced, and the accuracy of the target detection model can be greatly improved, thereby quickly completing data quality improvement.
[0055] The present invention can simultaneously achieve the dual purposes of automated inspection of target detection data and optimization of model accuracy, and can significantly improve labor efficiency and reduce labor costs.
[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0058] Figure 1 is a schematic diagram of a flow chart of an embodiment of the present invention;
[0059] Figure 2 It is a schematic diagram of the effect of an embodiment of the present invention. DETAILED DESCRIPTION
[0060] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0061] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0062] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0063] Embodiment 1
[0064] like Figure 1 As shown, an image annotation method comprises the following steps:
[0065] (1) Collect a batch of surveillance video data, about 90,000 data in this embodiment, and perform manual annotation D1. The total image list is represented by L.
[0066] (2) Use a target detection model A1 (the rtdetr model is used in this embodiment, and the specific target detection model is not required) and use the manually labeled data D1 for training. The training parameters of this training are as follows:
[0067] 1) In this embodiment, the number of training iterations is 72 epochs;
[0068] 2) In this embodiment, the training data is enhanced by -3 to +3 degree image rotation, translation, random flipping and other data enhancements.
[0069] 3) The learning rate is calculated according to the number of graphics cards. In this embodiment, 0.01 eight cards 2080ti are used for training.
[0070] 4) The pre-trained model for target detection uses a model pre-trained on the coco dataset.
[0071] (3) Use the trained target detection model A1 to perform forward prediction on the data list L, and the prediction result is Y1.
[0072] (4) Confidence learning-based evaluation scores are constructed for each image to check for annotation issues in target detection labels.
[0073] The confidence matrix is constructed as follows. Assume that the i-th annotation box in an image contains 5 values denoted as D i =(x d1 ,y d1 ,x d2 ,y d2 ,c d ), where x d1 is the horizontal coordinate of the upper left corner of the target bounding box, y d1 is the upper left corner ordinate of the target bounding box, x d2is the horizontal coordinate of the lower right corner of the target bounding box, y d2 is the ordinate of the lower right corner of the target bounding box, c d is the label value of the bounding box.
[0074] Assume that the i-th prediction box on an image contains 6 values, and the value of the i-th prediction box is represented by P i =(x p1 ,y p1 ,x p2 ,y p2 ,c p ,p), where x p1 is the horizontal coordinate of the upper left corner of the target bounding box, y p1 is the upper left corner ordinate of the target bounding box, x p2 is the horizontal coordinate of the lower right corner of the target bounding box, y p2 The ordinate of the lower right corner of the target bounding box, c p is the predicted label value, and p is the predicted probability value.
[0075] For an image, there are n annotations and m predictions.
[0076] 1) Construct the IoU matrix, and calculate the IoU values of n annotations and m predictions to obtain M iou Matrix, M iou The size is n×m. The IoU calculation method is as follows:
[0077]
[0078] 2) Calculate the distance matrix. For n annotations and m predictions, calculate the distances to obtain M dis Matrix, M dis The size of is n×m, and the distance is calculated as follows:
[0079]
[0080] where x dc ,y dc ,x pc ,y pc They are respectively the center point position of the i-th annotation box and the center point position of the j-th detection target.
[0081] 3) Calculate the similarity matrix M sim , the calculation method is as follows:
[0082] M sim =τ×M iou +(1-τ)×(1-M dis )
[0083] Where τ is the relaxation factor. The value in this experiment is 0.9.
[0084] 4) Establish M score Question score matrix, M score The size of is n×m.
[0085] aM iou The IoU is greater than 0.5, and c d =c p The index of M score The position is set to 0,
[0086] b. Medium IoU is greater than 0.5, c d ≠c p The index of M score The position is set to 1-p, where p is the predicted probability value.
[0087] c. Others The IoU is not greater than 0.5 in M score The position is set to 0 or 1-p (1-p is used when there is a prediction result).
[0088] 5) Find the final score
[0089]
[0090] Where E is represented by an m×n matrix with all 1s.
[0091] (5) Repeat step (4) to calculate the image score for each image in the L list.
[0092] (6) Set the screening threshold θ. If S>θ, the label is considered to have a problem. Otherwise, the label is not a problem. Here θ is set to 0.8. The problematic list is represented as I1, as Figure 2 An example of problematic image annotations and predictions, with blue being the predicted annotations and green being the manual annotations.
[0093] (7) Replace the labels of the I1 list in the manually annotated D1 with the prediction results W1 (W1 is the prediction result corresponding to the erroneous part of the image in Y1) as pseudo labels and perform a new round of iteration.
[0094] (8) Since the predictions in W1 may contain certain errors due to the training obtained from the previously labeled erroneous data, it is necessary to design an error correction penalty loss function to penalize the erroneous data training during the training process, thereby reducing the impact of erroneous data on training.
[0095]
[0096] Among them, Ls is the overall loss, Ls D is the loss of the manual annotation part that is considered to have no wrong annotations, Ls I It is the loss that is suspected to be wrong in manual annotation. is the penalty factor, which is set to 0.5 here. I1 is the list with problems in the data list, and L-I1 refers to the loss of all lists that are considered to have no wrong labels. * They all include the sum of three target detection losses, namely classification loss, bbox loss and IoU calculation loss. There is no specific restriction here. In this embodiment, the classification loss uses qualityfocalloss, the bbox loss uses L1 regression loss, and the IoU loss uses GIoU loss.
[0097] (9) Repeat (2)-(8) and continue to iterate the model until the number of lists I1 screened out in step (6) decreases to 0 or no longer decreases, and then train the last round.
[0098] Of course, in other embodiments, the above selections and parameter settings can be adjusted according to specific circumstances.
[0099] After simulation verification, in the first iteration, there were 10,177 images with missing or mislabeled labels that needed to be revised. After adding the error correction penalty loss function and continuing the iterative training, after the second inspection, a total of 5,135 images with missing or mislabeled labels were generated for revised data. After the third round of iterative training and inspection, no data that needed to be revised was generated.
[0100] The overall accuracy F1 value of the third round of model training increased by 11% compared to the first version of the model.
[0101] If manual verification is used to revise the 90,000 target detection data, it will take about three weeks, assuming that each person reviews 3,000 to 4,000 images per day and also needs to check the labels. By using the method in this article, three rounds of data checks can be completed in less than three weeks, which can quickly improve the data quality.
[0102] Embodiment 2
[0103] An image annotation system, comprising:
[0104] An acquisition module is configured to acquire monitoring data and manually annotated data of the monitoring data;
[0105] A training module is configured to train a target detection model using the manually annotated data;
[0106] A prediction module is configured to use the trained target detection model to perform forward prediction on the data list of the monitoring data to obtain a prediction result;
[0107] An evaluation module, configured to calculate an evaluation score for each image in the data list based on confidence learning, and use the evaluation score to check for existing labeling problems in the target detection label;
[0108] The judgment module is configured to consider that there is a problem with an image if the evaluation score of the image is greater than a set threshold, record the image with the problem, and form a new list;
[0109] An updating module is configured to replace the labels of the new list with the corresponding prediction results as pseudo labels, and perform a new round of iteration;
[0110] The iteration module is configured to return to call the training module until the number of new lists formed meets the iteration condition.
[0111] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0112] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0113] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0114] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.
Claims
1. An image annotation method, comprising the following steps: Using the acquired image data and the manually annotated data of the image data, training the target detection model; Its characteristic is that all image data are forward predicted using the trained target detection model to obtain corresponding prediction results; Calculate evaluation scores for all image data based on confidence learning; If the evaluation score of an image is greater than the set threshold, the image is considered to have problems, and all problematic images are recorded and a new list is formed; Replacing the label of the image in the new list with the prediction result corresponding to the image; Return to the step of training the target detection model and iterate until the number in the currently formed new list meets the set conditions.
2. The image annotation method according to claim 1, characterized in that: Before training the target detection model, the following steps are included: Set the number of training iterations; The training data is augmented by rotating, translating, and / or randomly flipping images within a set range.
3. The image annotation method according to claim 1, characterized in that: The specific process of calculating the evaluation scores of all image data based on confidence learning includes: For each image’s n annotations and m predictions, the IoU values are calculated to obtain the IoU matrix; For n annotations and m predictions, distance calculations are performed to obtain distance matrices; Based on the IoU matrix and the distance matrix, a similarity matrix is calculated; Based on the IoU matrix, establish a question data matrix; The data matrix and the similarity matrix are divided according to the question, and the final score is calculated.
4. The image annotation method according to claim 3, characterized in that: The calculation process of the IoU matrix includes: Among them, the value of the i-th annotation box is represented by D i =(x d1 ,y d1 ,x d2 ,y d2 ,c d ), where x d1 is the horizontal coordinate of the upper left corner of the target bounding box, y d1 is the upper left corner ordinate of the target bounding box, x d2 is the horizontal coordinate of the lower right corner of the target bounding box, y d2 is the lower right corner ordinate of the target bounding box, c d is the label value of the bounding box; The value of the i-th prediction box is represented as P i =(x p1 ,y p1 ,x p2 ,y p2 ,c p ,p), where x p1 is the horizontal coordinate of the upper left corner of the target bounding box, y p1 is the upper left corner ordinate of the target bounding box, x p2 is the horizontal coordinate of the lower right corner of the target bounding box, y p2 The ordinate of the lower right corner of the target bounding box, c p is the predicted label value, and p is the predicted probability value.
5. The image annotation method according to claim 3, characterized in that: In the distance matrix, the calculation process of each element includes: where x dc ,y dc ,x pc ,y pc They are respectively the center point position of the i-th annotation box and the center point position of the j-th detection target; Or, the similarity matrix is: M sim =τ×M iou +(1-τ)×(1-M dis ) Where τ is the relaxation factor, M iou is the IoU matrix, M dis is the distance matrix.
6. The image annotation method according to claim 5, wherein the question Score matrix M score for: If M IoU The IoU is greater than 0.5, and c d =c p The index of M score The position is set to 0; If IoU is greater than 0.5, c d ≠c p The index of M score The position is set to 1-p, where p is the predicted probability value. If IoU is not greater than 0.5, in M score The position is set to 0 or 1-p, and 1-p is used where there is a prediction result.
7. An image annotation method as claimed in claim 6, characterized in that: The final score is: Where E is represented by an m×n matrix with all 1s.
8. An image annotation method according to any one of claims 1 to 7, characterized in that: include: Construct an error correction penalty loss function and use it to punish erroneous data training during training; The error correction penalty loss function is: Among them, Ls is the overall loss, Ls D is the loss of the manual annotation part that is considered to have no wrong annotations, Ls I It is the loss that is suspected to be wrong in manual annotation. is the penalty factor, I1 is the problematic list in the data list, and L-I1 refers to the loss of all lists that are considered to have no incorrect annotations; Or further, each loss includes the sum of classification loss, bbox loss and IoU calculation loss.
9. The image annotation method according to claim 1, characterized in that: The set condition is that the value is less than the set value or does not decrease any more.
10. An image annotation system, comprising: A model training module is configured to train the target detection model using the acquired image data and the manually annotated data of the image data; Its characteristics include: A prediction module is configured to use the trained object detection model to perform forward prediction on all image data to obtain corresponding prediction results; An evaluation module is configured to calculate evaluation scores of all image data based on confidence learning; if the evaluation score of an image is greater than a set threshold, the image is considered to have a problem, all images with problems are recorded, and a new list is formed; A correction module, configured to replace the label of the image in the new list with the prediction result corresponding to the image; The iteration module is configured to call the model training module and iterate until the number of the new list currently formed meets the set conditions.