Image annotation review method, device, electronic device and storage medium

Abstract

The present application provides a method, apparatus, electronic device, and computer-readable storage medium for image annotation review. The method comprises: using a trained target detection model to perform target detection on a set of annotated images to be reviewed, obtaining a detection result for each image in the set; for each image in the set, removing targets whose confidence in the detection result is greater than a first threshold; sorting all images in the set according to the remaining targets in each image; and annotating and reviewing the set of images in the sorted order. The present application can solve the problem of low annotation quality or slow review speed caused by the large randomness of annotation review in the prior art, and can improve the annotation review speed and thus improve the annotation quality.

Description

Technical Field

[0001] The present application relates to the field of computer vision, specifically to target detection, intelligent auxiliary labeling and intelligent review technology, and especially to a method, device, electronic device and computer-readable storage medium for image labeling and review. Background Art

[0002] To train a highly accurate network model, the neural network needs to be fed with massive amounts of high-quality data. Taking object detection as an example, to ensure data quality, a large number of images need to be annotated. After the annotator completes the annotation of the image and submits it for review, the reviewer needs to review the submitted image. Generally, unqualified annotations include omissions, incorrect annotations, and inaccurate annotations. Traditional review methods include random sampling. When the failure rate of the sampled images reaches a certain threshold (usually 10%), the images are considered unqualified and need to be re-annotated. This method is highly random and difficult to guarantee the accuracy of the annotations. However, if all annotated images are reviewed, while the accuracy of the annotations can be improved, it will be very time-consuming. Summary of the Invention

[0003] The purpose of this application is to provide a method, device, electronic device and storage medium for image annotation review that can quickly pre-screen all unqualified images for reviewers to review first.

[0004] The purpose of this application is achieved by the following technical solutions:

[0005] In a first aspect, this application provides a method for reviewing image annotations, comprising:

[0006] Use the trained object detection model to detect the annotated image set to be reviewed, and obtain the object detection result of each image in the image set;

[0007] For each picture in the picture set, remove the target whose confidence in the detection result is greater than a first threshold;

[0008] sorting all images in the image set according to the remaining target of each image; and

[0009] The pictures in the picture set are labeled and reviewed in the sorted order.

[0010] The beneficial effect of this technical solution is that by using a trained model to detect the set of images to be reviewed and sorting them according to the detection results, it can quickly filter out images with unqualified annotations and put them in the front, which is convenient for auditors to review. Moreover, since there is no need to sample the annotations, the accuracy of the annotations can be greatly improved, saving time.

[0011] In some optional embodiments, sorting all images in the image collection based on the remaining targets in each image includes calculating the number of remaining targets in each image and sorting the images in descending order based on the number of remaining targets. This technical solution has the beneficial effect of sorting the images in descending order of the number of low-confidence detection results, which can place images with many unqualified annotations at the top, further saving review time and improving review efficiency.

[0012] In some optional embodiments, sorting all images in the image collection according to the remaining targets of each image includes calculating the sum of the loss values of the remaining targets for each image, and sorting the images in descending order based on the sum of the loss values. This technical solution has the beneficial effect of arranging the images in descending order of the sum of the loss values of the detection results, so that images with more and larger losses are ranked first, further saving review time and improving review efficiency.

[0013] In some optional embodiments, sorting all images in the image set according to the remaining targets of each image includes: calculating the number of the remaining targets of each image, and sorting them in descending order by the number to obtain a first sorting of the images; calculating the sum of the loss values of the remaining targets of each image, and sorting them in descending order by the sum of the loss values to obtain a second sorting of the images; and merging the first sorting with the second sorting in proportion to obtain a final sorting of the images. The beneficial effect of this technical solution is that, first, the images are sorted in descending order by the number of detection results with low confidence, so that images with many unqualified labels can be placed in front, and then the images are arranged in descending order by the sum of the losses of the detection results, so that images with many and large losses are placed in front. After the two sortings are merged in proportion, the images can be sorted by comprehensively considering both the confidence and loss factors, further improving the accuracy of the sorting, thereby further saving the review time and improving the efficiency of the review.

[0014] In some optional embodiments, when using the trained model to test the annotated image set for review, only the losses of the images with annotated detection targets are backpropagated. This technical solution has the beneficial effect of removing images that have a negative impact on model training, that is, reducing the number of negative samples caused by images without detection targets during the prediction process, thereby improving the model's detection performance for the target to be predicted, and also improving the detection performance of images with unqualified annotations.

[0015] In some optional embodiments, the first threshold is 0.9. The beneficial effect of this technical solution is that by removing objects with a confidence level greater than 0.9, qualified annotations are basically excluded, thereby preliminarily screening out unqualified annotations and their corresponding images, thereby improving review efficiency.

[0016] In some optional embodiments, the first and second sortings are combined in a 1:1 ratio to obtain a final sorting of the images. This technical solution has the beneficial effect of combining the two sortings in a 1:1 ratio to obtain a final sorting of all images, which can comprehensively consider various situations including missing labels, incorrect labels, and inaccurate labels, thereby obtaining a more accurate sorting. Specifically, images with unqualified labels are sorted to the front as much as possible to facilitate subsequent review.

[0017] In some optional embodiments, for each image in the image collection, images containing objects with a confidence level less than a second threshold in the detection results are extracted as unqualified images. This technical solution has the beneficial effect of excluding images with objects with a confidence level less than a second threshold from being labeled as unqualified. This technical solution has the beneficial effect of expediting the review of annotations by auditors by setting a second threshold to expedite the review of annotations, as it indicates that the model cannot fit the annotations.

[0018] In a second aspect, the present application provides a device for reviewing image annotations, comprising:

[0019] Detection module: Use the trained object detection model to detect the annotated image set to be reviewed, and obtain the object detection results for each image in the image set;

[0020] A screening module: for each picture in the picture set, removing objects in the detection results whose confidence level is greater than a first threshold;

[0021] Sorting module: sorting all images in the image set according to the remaining target of each image: and

[0022] Review module: annotate and review the pictures in the picture set in the sorted order.

[0023] In some optional embodiments, in the sorting module, the number of the remaining objects in each picture is calculated, and the pictures are sorted in descending order according to the number.

[0024] In some optional embodiments, in the sorting module, the sum of the loss values of the remaining targets of each image is calculated, and the images are sorted in descending order according to the size of the sum of the loss values.

[0025] In some optional embodiments, in the sorting module, the number of the remaining targets of each image is calculated and sorted in descending order by the number to obtain a first sorting of the images; the sum of the loss values of the remaining targets of each image is calculated and sorted in descending order by the sum of the loss values to obtain a second sorting of the images; and the first sorting and the second sorting are proportionally merged to obtain a final sorting of the images.

[0026] In some optional embodiments, in the detection module, when the trained model is used to detect the annotated set of images to be reviewed, only the losses of the images with annotated detection targets are back-propagated.

[0027] In some optional embodiments, in the screening module, the first threshold is 0.9.

[0028] In some optional embodiments, in the sorting module, the first sorting and the second sorting are merged in a 1:1 ratio to obtain the final sorting of the pictures.

[0029] In some optional embodiments, in the screening module, for each picture in the picture set, pictures containing targets with a confidence level less than a second threshold in the detection results are extracted as labeled unqualified pictures.

[0030] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0031] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above methods are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The present application is further described below with reference to the accompanying drawings and examples.

[0033] Figure 1 This is a flowchart of the first embodiment of the image annotation review method according to the present application;

[0034] Figure 2 is a flowchart of a second embodiment of the image annotation review method according to the present application;

[0035] Figure 3 is a flowchart of a third embodiment of the image annotation review method according to the present application;

[0036] Figure 4 It is a structural diagram of a picture annotation review device according to the present application;

[0037] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0038] Figure 6 It is a structural diagram of a program product for implementing a method for image annotation and review provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0040] See also Figure 1 The first embodiment of the present application provides a method for image annotation review, which generally includes steps S1 to S4.

[0041] Step S1: Use the trained model to detect the annotated image set to be reviewed, and obtain the detection results of each image in the image set.

[0042] The trained target detection model is a target detection model that has been pre-trained through qualified intelligent annotation and is used to detect one or more targets in a picture. The trained target detection model is used to detect the annotated picture in order to fit the annotation. Annotations that cannot be fitted indicate problems with the annotation. During the detection process, only the loss of the picture with the annotated detection target is back-propagated during back propagation. This can remove pictures that have a negative effect on model training due to the absence of detection targets (for example, no annotation box), that is, reduce the negative samples caused by pictures without detection targets during the prediction process, thereby improving the model's detection performance for the target to be predicted, and also improving the detection performance for pictures with unqualified annotations.

[0043] S2: For each picture in the picture set, remove the objects in the detection results whose confidence level is greater than a first threshold.

[0044] The trained model detects the annotated image and outputs a detection result with confidence. There may be multiple targets in an image, and the annotation quality of each target may be different. Therefore, in the second step, the targets with good annotation quality and good detection by the model are first removed by removing targets with high confidence. For example, the first threshold is 0.9. After removing targets with confidence greater than 0.9, the remaining targets are likely to be unqualified targets.

[0045] For example, four targets are detected in an image with confidence levels of 0.95, 0.90, 0.78, and 0.46, respectively. After removing targets with confidence levels of 0.95 and 0.90, two targets with confidence levels of 0.78 and 0.46 remain.

[0046] S3: Sort all pictures in the picture set according to the remaining targets of each picture: calculate the number of the remaining targets of each picture, and sort the pictures in descending order according to the number.

[0047] The remaining objects that are likely to be labeled as unqualified are sorted according to a certain evaluation standard. The quality of the evaluation standard directly affects the final sorting result. In this embodiment, the image is sorted in descending order based on the number of remaining objects (for example, objects with a confidence score less than 0.9). Therefore, images with more remaining objects are ranked first as images with a higher probability of being labeled as unqualified, and are reviewed first in subsequent steps.

[0048] For example, in the above-mentioned picture, there are two targets with remaining confidences of 0.78 and 0.46, so the number of remaining targets in this picture is 2. If there are pictures with a number of remaining targets greater than 2, the pictures with a number of remaining targets greater than 2 are ranked higher.

[0049] S4: marking and reviewing the sorted picture set according to the arrangement order of the pictures.

[0050] Since images with more targets and low confidence levels were placed at the front in the previous step, images with a high probability of unqualified labeling are prioritized for review in this step according to the ranking in the previous step, thereby speeding up the reviewer's review of the standards and helping to improve the quality of image labeling.

[0051] Figure 2 The image standard review method according to the second embodiment of the present application is shown. The image standard review method according to the second embodiment differs from the image standard review method according to the first embodiment only in that, in step S3', when sorting all images in the image set according to the remaining targets of each image, the sum of the losses of the remaining targets of each image is calculated, and the images are sorted in descending order according to the size of the sum of the losses.

[0052] In this embodiment, because the loss of the trained model will be very large for targets that are not labeled properly, the larger the sum of the loss values of the targets in an image, the greater the possibility that the image is not labeled properly. Therefore, the images are sorted in descending order based on the sum of the losses of the remaining targets in each image. This can put the unqualified images in front as much as possible, so that they will be reviewed first in subsequent steps, thereby improving the review efficiency and helping to improve the quality of image labeling.

[0053] Figure 3 The picture standard review method according to the third embodiment of the present application is shown. The picture standard review method according to the third embodiment is different from the picture standard review method according to the first embodiment or the second embodiment only in that, in step S3", when sorting all the pictures in the picture set according to the remaining targets of each picture, first calculate the number of the remaining targets of each picture, and sort them in descending order according to the size of the number to obtain the first sorting of the pictures; then calculate the sum of the losses of the remaining targets of each picture, and sort them in descending order according to the size of the sum of the losses to obtain the second sorting of the pictures; finally, merge the first sorting and the second sorting in proportion to obtain the final sorting of the pictures. Preferably, for example, the first sorting and the second sorting are merged in a ratio of 1:1 to obtain the final sorting of the pictures.

[0054] In this embodiment, various factors of the detection results, namely the number of low-confidence targets detected and the sum of the loss, are comprehensively considered to sort the images. This sorting can place images with higher failure rates at the top, thereby further improving review efficiency and, in turn, annotation quality.

[0055] The image annotation review method of this application is not limited to the above three embodiments. For example, in a preferred embodiment, after the model test is completed, for each image in the image collection, images containing targets with a confidence level less than a second threshold in the test results can be first selected as unqualified images for priority review. The second threshold can be, for example, 0.1. In this way, images most likely to be unqualified can be pre-reviewed, improving review efficiency.

[0056] Note that the values of the first threshold and the second threshold mentioned above are only examples. The values of the first threshold and the second threshold can be set arbitrarily according to actual conditions, as long as they can filter out qualified annotations or filter out the annotations that are most likely to be unqualified.

[0057] In addition, the evaluation criteria used to sort the pictures are not limited to the criteria described in the above embodiment. Any sorting criteria can be used as long as the pictures marked as unqualified are sorted at the front.

[0058] The above describes the image annotation review method according to the present application.

[0059] By using the image annotation review method according to the present application, the trained model is first used to detect the image set to be reviewed, and then sorted according to the detection results. The images with annotation problems can be placed at the front, thereby improving the speed of subsequent annotation review. Moreover, since there is no need to sample the annotations, but to screen out the images with annotation problems, the accuracy of the annotation can be greatly improved, the quality of the annotation can be improved, and time can be saved.

[0060] See also Figure 4 The embodiment of the present application also provides a picture annotation and review device, the specific implementation method of which is consistent with the implementation method and the technical effect achieved in the embodiment of the above method, and some contents will not be repeated here.

[0061] The device comprises:

[0062] Detection module 101: Use the trained object detection model to detect the annotated image set to be reviewed, and obtain the object detection result of each image in the image set;

[0063] Screening module 102: for each picture in the picture set, removing objects in the detection results whose confidence level is greater than a first threshold;

[0064] Sorting module 103: sorting all pictures in the picture set according to the remaining target of each picture; and

[0065] Review module 104: displays the pictures in the sorted order for annotation review.

[0066] In a specific embodiment, the sorting method performed in the sorting module 103 includes the methods described in the first to third embodiments above.

[0067] The above describes the image annotation and review device according to the present application.

[0068] See also Figure 5 , an embodiment of the present application further provides an electronic device 200, which includes at least one memory 210, at least one processor 220, and a bus 230 connecting different platform systems.

[0069] The memory 210 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 211 and / or a cache memory 212 , and may further include a read-only memory (ROM) 213 .

[0070] Among them, the memory 210 also stores a computer program, which can be executed by the processor 220, so that the processor 220 performs the steps of the real-time video processing method in the embodiment of the present application. Its specific implementation method is consistent with the implementation method and the technical effect achieved in the embodiment of the above-mentioned real-time video processing method, and some contents will not be repeated here.

[0071] Memory 210 may also include a utility 214 having at least one program module 215, such program modules 215 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0072] Accordingly, the processor 220 may execute the aforementioned computer program and the utility 214 .

[0073] The bus 230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0074] The electronic device 200 may also communicate with one or more external devices 240, such as a keyboard, pointing device, Bluetooth device, etc., and may also communicate with one or more devices capable of interacting with the electronic device 200, and / or any device that enables the electronic device 200 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may be performed via an input / output interface 250. Furthermore, the electronic device 200 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 260. The network adapter 260 may communicate with other modules of the electronic device 200 via the bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 200, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0075] The embodiment of the present application also provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed, it implements the steps of the real-time video processing method in the embodiment of the present application. Its specific implementation method is consistent with the implementation method and technical effects recorded in the embodiment of the above-mentioned real-time video processing method, and some contents are not repeated here.

[0076] Figure 6The program product 300 provided in this embodiment for implementing the above-mentioned real-time video processing method is shown. It can be implemented in a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product 300 of the present invention is not limited to this. In this application, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device. The program product 300 can utilize any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0077] A computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, or any suitable combination thereof. The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and conventional procedural programming languages such as C or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0078] This application is explained from the perspectives of purpose of use, effectiveness, progress and novelty, and has complied with the functional enhancement and use requirements emphasized by the Patent Law. The above description and drawings of this application are only preferred embodiments of this application and are not intended to limit this application. Therefore, all structures, devices, features, etc. that are similar or identical to those of this application, that is, all equivalent replacements or modifications made in accordance with the scope of the patent application of this application, should fall within the scope of protection of the patent application of this application.

Claims

1. A method for reviewing image annotation, characterized in that: include: Use the trained object recognition model to detect the annotated image set to be reviewed, and obtain the object detection result of each image in the image set; For each picture in the picture set, remove the target whose confidence in the detection result is greater than a first threshold; Sorting all images in the image set according to the remaining target of each image, including: Calculating the number of the remaining objects in each image and sorting them in descending order by number to obtain a first sorting of the images; calculating the sum of the loss values of the remaining objects in each image and sorting them in descending order by the sum of the loss values to obtain a second sorting of the images; and merging the first sorting and the second sorting in proportion to obtain a final sorting of the images; and annotating and reviewing the pictures in the picture set in the sorted order.

2. The image annotation review method according to claim 1, characterized in that: Sorting all the pictures in the picture set according to the remaining target of each picture includes: Calculate the number of the remaining objects in each picture, and sort them in descending order according to the number; or, Calculate the sum of the loss values of the remaining targets for each image, and sort them in descending order according to the sum of the loss values.

3. The image annotation review method according to any one of claims 1 to 2, characterized in that: in, When using the trained model to detect the annotated set of images to be reviewed, only the losses of the images with annotated detection targets are back-propagated.

4. The image annotation review method according to any one of claims 1 to 2, characterized in that: in, The first threshold is 0.

9.

5. The image annotation and review method according to claim 1, characterized in that: in, The first sorting and the second sorting are merged in a 1:1 ratio to obtain a final sorting of the pictures.

6. The image annotation and review method according to any one of claims 1 to 2, characterized in that: in, For each picture in the picture set, pictures containing targets with a confidence level less than a second threshold in the detection results are extracted as labeled unqualified pictures.

7. A device for reviewing image annotations, characterized in that: include: Detection module: Use the trained object detection model to detect the annotated image set to be reviewed, and obtain the object detection results for each image in the image set; A screening module: for each picture in the picture set, removing objects in the detection results whose confidence level is greater than a first threshold; A sorting module is configured to sort all images in the image set according to the remaining targets of each image, including: calculating the number of the remaining targets of each image and sorting them in descending order by the number to obtain a first sorting of the images; calculating the sum of the loss values of the remaining targets of each image and sorting them in descending order by the sum of the loss values to obtain a second sorting of the images; and merging the first sorting and the second sorting in proportion to obtain a final sorting of the images; as well as Review module: annotate and review the pictures in the picture set in the sorted order.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that A computer program is stored, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images