Data annotation quality detection method and device and data annotation quality verification method and device
By automatically detecting image data quality using a target detection model, the problem of low accuracy and low efficiency of manual verification is solved, enabling fast and accurate data annotation quality detection and verification.
Patent Information
- Application Number
- CN202511051258.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-29
- Publication Date
- 2025-11-18
AI Technical Summary
When manually verifying the quality of data annotations, the accuracy and efficiency are low, especially in the case of large batches of multi-category annotated data, where manual review is difficult to quickly and accurately verify erroneous annotated data.
By using a pre-trained object detection model, data quality is automatically detected by extracting current and historical annotation features of images, including correct, incorrect, and missed detection results. The model parameters are then updated using a loss function to achieve fast and accurate data quality detection.
It improves the accuracy and efficiency of data annotation quality detection, reduces erroneous annotations caused by human fatigue, and enhances the efficiency of manual verification.
Smart Images
Figure CN120976129A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of target detection technology, specifically to a method for detecting data annotation quality, a method for verifying data annotation quality, and an apparatus. Background Technology
[0002] In image detection, a large amount of manual labeling of images to their categories is typically required before they are input into object detection models for detection, thus obtaining the labeling quality detection results of the image data. However, in the field of vision, to fully utilize the power of large models and enable them to recognize everything, a large amount of labeled data of different categories is needed.
[0003] Currently, the labeled data is not only large in volume, but also involves many image types, such as more than a thousand categories, or even tens of thousands. Labeling such data, even with pre-labeling methods, requires manual review to ensure label quality. However, labeling so many categories results in densely packed bounding boxes on the images, severely hindering the efficiency of manual review and correction. In addition, when the number and types of manually labeled images are large, errors in labeled data are inevitable due to long working hours, and humans cannot quickly and accurately verify batches of erroneous samples. Summary of the Invention
[0004] In view of this, the present invention provides a method for detecting data annotation quality, a method for verifying data annotation quality, and an apparatus to solve the problems of low accuracy and low efficiency of manual verification.
[0005] In a first aspect, the present invention provides a method for detecting the quality of data annotation, the method comprising:
[0006] Obtain multiple target images and the current annotation results for each target image;
[0007] Extract the current annotation features from the current annotation results of each target image. The current annotation features include correct annotations and incorrect annotations.
[0008] The current labeled features extracted from each target image are input into the target detection model for detection, and the corresponding data quality detection results for each target image are obtained. The target detection model is a pre-trained neural network, and the data quality detection results for each target image include: correct detection results, incorrect detection results, and missed detection results.
[0009] The data labeling quality detection method in this embodiment utilizes a target detection network to detect multiple target images in batches, thereby quickly and accurately obtaining the data quality detection results for each target image. Therefore, it eliminates the need for manual detection of large batches of target images of different categories one by one, thus avoiding the problem of incorrect labeling of target image categories due to human fatigue, which affects the data category labeling quality. Furthermore, the method in this embodiment is more efficient than the method of manually labeling target image categories.
[0010] In one optional implementation, the data annotation quality detection method in this disclosure includes:
[0011] Obtain multiple historical images and the historical annotation results for each historical image;
[0012] Historical annotation features are extracted from the historical annotation results of each historical image. These historical annotation features include correct annotation markers and incorrect annotation markers.
[0013] Historical annotation features are input into the target detection model for training, generating data quality detection results for each historical image, as well as the loss between the historical annotation results and the data quality detection results for each historical image. The loss is used to update the parameter values of the target detection model. The data quality detection results for each historical image include: correct detection results, incorrect detection results, and missed detection results.
[0014] The above-described implementation methods in this disclosure are beneficial for improving the accuracy of the target detection model.
[0015] In one optional implementation, the data annotation quality detection method in this disclosure includes:
[0016] Use all historical images as the test set;
[0017] Obtain the average category accuracy for each historical image in the test set;
[0018] If the average class accuracy of each historical image in the test set is greater than or equal to the preset threshold, then the class labeling quality of multiple historical images is determined to meet the requirements.
[0019] If the average class accuracy of each historical image in the test set is not greater than a preset threshold, then the class labeling quality of multiple historical images is determined to be unsatisfactory.
[0020] In one alternative implementation, the average class accuracy for each historical image in the test set is obtained, including:
[0021] Extract the current annotation features from the current annotation results of each historical image in the test set;
[0022] The current annotation features extracted from the current annotation results of each historical image in the test set are input into the target detection model for training, which corresponds to the loss between the data quality detection result of each historical image in the test set and the current annotation result of that historical image.
[0023] The average class accuracy of each historical image in the test set is calculated based on the loss between the data quality detection result of each historical image in the test set and the current annotation result of that historical image.
[0024] Calculate the average class accuracy for each historical image in the test set based on the average class accuracy for each historical image in the test set.
[0025] This disclosure embodiment can accurately determine the data quality of each target image of different types by obtaining the average category accuracy of each target image in the test set.
[0026] In one optional implementation, the data annotation quality detection method in this disclosure includes:
[0027] Calculate the recall and precision for each historical image in the test set based on the data quality detection result for each historical image and the loss between the current annotation result of that historical image and the data quality detection result of that historical image.
[0028] In one optional implementation, the data annotation quality detection method in this disclosure includes:
[0029] Based on the data quality detection result of each historical image and the loss between the current annotation result of that historical image, the sum of the losses of multiple historical images is calculated;
[0030] Calculate the average of the loss sums from multiple historical images;
[0031] The average of the sum of losses from multiple historical images is used as the loss for multiple target images.
[0032] The embodiments of this disclosure can accurately obtain the loss of multiple target images through the above method.
[0033] Secondly, this embodiment provides a method for verifying the quality of data annotation, including:
[0034] The target image corresponding to the error detection result is obtained by using a data annotation quality detection method;
[0035] Verify the actual annotation information of the target image corresponding to the error detection results.
[0036] The target image corresponding to the error detection result is obtained by verifying the data annotation quality using the above-mentioned method.
[0037] Thirdly, the present invention provides a data annotation quality detection device, the device comprising:
[0038] The acquisition module is used to acquire multiple target images and the current annotation results for each target image;
[0039] The extraction module is used to extract the current annotation features from the current annotation results of each target image. The current annotation features include correct annotation labels and incorrect annotation labels.
[0040] The detection module is used to input the current labeled features extracted from each target image into the target detection model for detection, and to detect the data quality of each target image. The target detection model is a pre-trained neural network, and the data quality detection results of each target image include: correct detection results, incorrect detection results, and missed detection results.
[0041] Fourthly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the data annotation quality detection method of the first aspect or each corresponding embodiment described above, or to perform the data annotation quality verification method of the second aspect or each corresponding embodiment described above.
[0042] Fifthly, the present invention provides a computer-readable storage medium storing computer instructions, which are used to cause a computer to perform the data annotation quality detection method of the first aspect or each of the corresponding embodiments described above, or to perform the data annotation quality verification method of the second aspect or each of the corresponding embodiments described above.
[0043] In a sixth aspect, the present invention provides a computer program product, including computer instructions, which are used to cause a computer to execute the data annotation quality detection method of the first aspect or each of the corresponding embodiments described above, or to execute the data annotation quality verification method of the second aspect or each of the corresponding embodiments described above. Attached Figure Description
[0044] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0045] Figure 1This is a flowchart illustrating a method for detecting the quality of data annotation according to an embodiment of the present invention;
[0046] Figure 2 This is a flowchart of a data annotation quality detection method according to an embodiment of the present invention;
[0047] Figure 3 This is a flowchart illustrating a method for verifying the quality of data annotation according to an embodiment of the present invention.
[0048] Figure 4 This is a simplified flowchart of a data annotation quality detection method according to an embodiment of the present invention;
[0049] Figure 5 This is a structural block diagram of a data annotation quality detection device according to an embodiment of the present invention;
[0050] Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0052] According to an embodiment of the present invention, an embodiment of a method for detecting the quality of data annotation is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0053] This embodiment provides a method for detecting the quality of data annotation, which can be used on mobile terminals such as mobile phones and tablets. Figure 1 This is a flowchart of a data annotation quality detection method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:
[0054] Step S101: Obtain multiple target images and the current annotation result for each target image.
[0055] Specifically, multiple target images can be batch detection data. The target image refers to the object of interest during detection. The target image can include, but is not limited to, detection images in autonomous driving scenarios, detection images of different landscapes and buildings in aerial photography scenarios, and detection images of different people's live streaming dynamics in live streaming scenarios. For example, in the above-mentioned autonomous driving scenario, the target image can be the detection vehicle at different driving stages, and in the above-mentioned aerial photography scenario, the target image can be different landscapes and buildings.
[0056] The current annotation results for each target image mentioned above include: correct annotation results and incorrect annotation results.
[0057] Step S102: Extract the current annotation features from the current annotation results of each target image. The current annotation features include correct annotation labels and incorrect annotation labels.
[0058] Specifically, the current annotation features can characterize the attribute information of the current annotation result of each target image. For example, the attribute information of the current annotation result of each target image can be identification information, which can include numerical identifiers, such as using 1 to identify a correct annotation result and using 0 to identify an incorrect annotation result. The identification information can also include letter identifiers, such as using A to identify a correctly annotated image type and using B to identify a correctly annotated image type.
[0059] Step S103: Input the current labeled features extracted from each target image into the target detection model for detection, and detect the data quality of each target image. The target detection model is a pre-trained neural network. The data quality detection results of each target image include: correct detection results, incorrect detection results, and missed detection results.
[0060] Specifically, the object detection model can be the YOLO model (You Only Look Once). Experiments show that YOLOv8 is more suitable, as it employs an advanced backbone network and neck architecture, achieving improved feature extraction and object detection performance. It focuses on maintaining the optimal balance between accuracy and speed, making it suitable for real-time object detection tasks in various application domains. This YOLO model can detect the true class, bounding box location, and bounding box size of each object image.
[0061] Table 1 below shows a comparison between the actual annotation and the actual detection results for each target image.
[0062] Table 1
[0063]
[0064] Table 1 above compares the category data quality detection results (correct detection results, incorrect detection results, and missed detection results) for each target image with the actual annotation of each target image. Figure 1 In this context, TP represents the number of target images of the correctly detected category, FN represents the number of target images that were missed, FP represents the number of target images of the falsely detected category, and TN represents the case where no target image was detected.
[0065] This embodiment acquires multiple target images and obtains the current annotation result for each target image; extracts current annotation features from the current annotation result of each target image, the current annotation features including correct annotation markers and incorrect annotation markers; inputs the extracted current annotation features of each target image into a target detection model for detection, corresponding to the data quality detection result of each target image. Therefore, this embodiment eliminates the need for manual detection of large batches of target images of different categories one by one, thereby avoiding the problem of incorrectly labeling target image categories due to manual fatigue, which affects the data category annotation quality. Furthermore, the method of this disclosure is more efficient than manually labeling target image categories.
[0066] This embodiment provides a method for detecting the quality of data annotation, which can be used on mobile terminals such as mobile phones and tablets. Figure 2 This is a flowchart of a data annotation quality detection method according to an embodiment of the present invention, such as... Figure 2 As shown, before step S101 of acquiring multiple target images, the following steps are also included:
[0067] Step S201: Obtain multiple historical images and the historical annotation results for each historical image.
[0068] Specifically, similarly, multiple historical images can be batch detection data. Historical images refer to the objects of interest during detection. These historical images can include, but are not limited to, detection images in autonomous driving scenarios, detection images of different landscapes and buildings in aerial photography scenarios, and detection images of different people's live streaming dynamics in live streaming scenarios. For example, for the aforementioned autonomous driving scenario, the historical images can be detection vehicles at different driving stages, and for the aforementioned aerial photography scenario, the historical images can be different landscapes and buildings.
[0069] When training an object detection model using a batch of historical images, for example, 1000 historical images, the current annotation result of each of these 1000 historical images is obtained. The current annotation result of each object image includes both correctly annotated and incorrectly annotated results.
[0070] Step S202: Extract historical annotation features from the historical annotation results of each historical image. The historical annotation features include correct annotation markers and incorrect annotation markers.
[0071] The current annotation features can characterize the attribute information of the current annotation result of each target image. For example, the attribute information of the current annotation result of each target image can be identification information. The identification information can include numerical identifiers, for example, using 1 to indicate a correct annotation result and using 0 to indicate an incorrect annotation result. The identification information can also include letter identifiers, for example, using A to indicate a correct annotation result and using B to indicate an incorrect annotation result.
[0072] Step S203: Input the historical annotation features into the target detection model for training, generate the data quality detection result of each historical image, and the loss between the historical annotation result and the data quality detection result of each historical image. The loss is used to update the parameter values of the target detection model. The data quality detection result of each historical image includes: correct detection result, incorrect detection result, and missed detection result.
[0073] In some optional implementations, the data annotation quality detection method in this disclosure includes:
[0074] Step a1: Select a portion of historical images from multiple historical images as the test set.
[0075] Specifically, for example, there are 1000 historical images, and these 1000 historical images are used as the test set.
[0076] Step a2: Obtain the average category accuracy for each historical image in the test set.
[0077] In some optional implementations, the average category accuracy for each historical image in the test set is obtained, including:
[0078] Step a21: Extract the current annotation features from the current annotation results of each historical image in the test set.
[0079] Step a22: Input the current annotation features extracted from the current annotation results of each historical image in the test set into the target detection model for training, and predict the data quality detection result of each historical image in the test set and the loss between the historical image.
[0080] The training methods for steps a21-a22 are the same as those for steps S101-S103.
[0081] Step a23: Calculate the average category accuracy for each historical image in the test set based on the data quality detection results and the loss of each historical image.
[0082] In some optional implementations, the average category accuracy for each historical image in the test set is calculated using the following formula:
[0083] AP=∫0 1 p(r)dr
[0084] Where p(r) represents the accuracy of each class of historical images, and AP represents the average accuracy of each class of historical images.
[0085] Step a24: Calculate the average class accuracy of each historical image in the test set based on the average class accuracy of each historical image in the test set.
[0086] In some optional implementations, the average class accuracy for each historical image in the test set is calculated using the following formula:
[0087]
[0088] Where mAP represents the average class accuracy for each historical image, c is the number of classes, and AP represents the average class accuracy for any historical image.
[0089] Step a3: If the average class accuracy of each historical image in the test set is greater than or equal to the preset threshold, then the class labeling quality of multiple historical images is determined to meet the requirements.
[0090] Step a4: If the average class accuracy of each historical image in the test set is not greater than the preset threshold, then the class labeling quality of multiple historical images is determined to be unsatisfactory.
[0091] Given the training and testing data mentioned above, if the class labeling quality of the training data is not high, resulting in a low mAP metric in the test, the overall quality of the class labeling of the historical images can be judged by calculating the average class accuracy of each historical image.
[0092] In some optional implementations, the data annotation quality detection method in this disclosure further includes: calculating the recall and precision of each historical image in the test set based on the loss between the data quality detection result of each historical image in the test set and the data quality detection result of the historical image.
[0093] Specifically, the recall rate for each historical image in the test set is calculated using the following formula, based on the loss between the data quality detection result of each historical image and the data quality detection result of that historical image:
[0094]
[0095] Where TP represents the number of historical images that were correctly detected, FN represents the number of historical images that were missed, and R represents the recall rate.
[0096] The recall rate mentioned above describes the ability of an object detection model to correctly detect objects. The larger the value, the stronger the model's ability to detect objects.
[0097] Specifically, the accuracy of each historical image in the test set is calculated based on the loss between the data quality detection result of each historical image and the data quality detection result of that historical image, using the following formula:
[0098]
[0099] Where TP represents the number of historical images that correctly detected the category, FP represents the number of historical images that falsely detected the category, and P represents the accuracy.
[0100] Accuracy describes the ability of an object detection model to correctly detect objects. The higher the value, the stronger the model's ability to detect objects.
[0101] Precision and recall are contradictory; a high recall rate generally leads to a lower precision rate.
[0102] In some optional implementations, the data annotation quality detection method in this disclosure includes:
[0103] Based on the loss between the data quality detection result of each historical image and the current annotation result of that historical image, the sum of the losses of multiple historical images is calculated.
[0104] Calculate the average of the loss sums from multiple historical images;
[0105] The average of the sum of losses from multiple historical images is used as the loss for multiple historical images.
[0106] This embodiment of the disclosure calculates the loss of multiple historical images as described above. Using this index parameter, in order to quickly verify the data that has been erroneously detected, for example, for data with erroneous detection results, the target boxes with larger losses are screened and corrected in a focused and detailed manner; for data with correct detection results and target boxes with smaller losses, they can be quickly verified once.
[0107] This embodiment provides a method for verifying the quality of data annotation, which can be used on mobile terminals such as mobile phones and tablets. Figure 3 This is a flowchart of a data annotation quality verification method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:
[0108] Step S301: Obtain the target image corresponding to the error detection result through the data annotation quality detection method.
[0109] Step S302: Verify the actual annotation information of the target image corresponding to the error detection result.
[0110] Specifically, such as Figure 4 As shown, for example, suppose we have a batch of labeled but unverified target images. We need to select a suitable target detection model (the model should not be too large or too small; too large a model is prone to overfitting, and too small a model is prone to underfitting; the preferred target detection model is YOLO). V 8) Train an object detection model using multiple target images. After training, test the trained object detection model on the aforementioned target images. Based on the test results, judge the overall annotation quality and the annotation quality of each category. Focus on verifying categories with substandard annotation quality (incorrect detection results). For categories with very high scores (correct detection results), verification may not be necessary, thereby improving the effectiveness of manual verification. Figure 1 As shown. At the same time, steps S101-S103 above can select the labels whose actual labels and predicted categories are inconsistent, and provide them to relevant personnel for further verification and correction. Therefore, the above can be further verified by manual verification to target the actual category of the target image corresponding to the erroneous detection result.
[0111] The data annotation quality verification method in this embodiment predicts the data quality detection result of each target image through the data annotation quality detection method. That is, it selects target images with substandard data quality detection results from the batch of target images and further focuses on verifying these target images with substandard data quality detection results. This not only makes up for the problem of incorrect annotation of target images due to human fatigue, but also improves the efficiency of manual verification.
[0112] This embodiment also provides a data annotation quality detection device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as already described. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0113] This embodiment provides a device for detecting the quality of data annotation, such as... Figure 5 As shown, it includes:
[0114] The first acquisition module 501 is used to acquire multiple target images and the current annotation result of each target image;
[0115] The second extraction module 502 is used to extract current annotation features from the current annotation results of each target image. The current annotation features include correct annotation labels and incorrect annotation labels.
[0116] The data detection module 503 is used to input the current labeled features extracted from each target image into the target detection model for detection, and to detect the data quality detection results of each target image. The target detection model is a pre-trained neural network, and the data quality detection results of each target image include: correct detection results, incorrect detection results, and missed detection results.
[0117] In some optional implementations, the data annotation quality detection device in this disclosure embodiment further includes:
[0118] The second acquisition module is used to acquire multiple historical images and the historical annotation results of each historical image;
[0119] The second extraction module is used to extract historical annotation features from the historical annotation results of each historical image. The historical annotation features include correct annotation markers and incorrect annotation markers.
[0120] The data training module is used to input historical annotation features into the target detection model for training, generate the data quality detection result of each historical image and the loss between the historical annotation result and the data quality detection result of each historical image. The loss is used to update the parameter values of the target detection model. The data quality detection result of each historical image includes: correct detection result, incorrect detection result, and missed detection result.
[0121] In one optional implementation, the data annotation quality detection device in this disclosure includes:
[0122] The data determination module is used to select all the multiple historical images as the test set.
[0123] The third acquisition module is used to obtain the average category accuracy for each historical image in the test set.
[0124] The first determining module is used to determine that the class labeling quality of multiple historical images meets the requirements if the average class accuracy of each historical image in the test set is greater than or equal to a preset threshold.
[0125] The second determination module is used to determine that the class labeling quality of multiple historical images does not meet the requirements if the average class accuracy of each historical image in the test set is not greater than a preset threshold.
[0126] In one optional implementation, the third acquisition module includes:
[0127] The feature extraction submodule is used to extract the current annotation features from the current annotation results of each historical image in the test set;
[0128] The feature training submodule is used to input the current annotation features extracted from the current annotation results of each historical image in the test set into the target detection model for training, and to predict the data quality detection result of each historical image in the test set and the loss between the historical annotation result of each historical image and the data quality detection result of each historical image.
[0129] The first calculation submodule is used to calculate the average category accuracy of each historical image in the test set based on the loss between the data quality detection result of each historical image in the test set and the current annotation result of the historical image.
[0130] The second calculation submodule is used to calculate the average class average accuracy of each historical image in the test set based on the average class average accuracy of each historical image in the test set.
[0131] In some optional implementations, the data annotation quality detection method in this embodiment includes:
[0132] The data calculation module is used to calculate the recall and precision of each historical image in the test set based on the loss between the data quality detection result of each historical image and the current annotation result of that historical image.
[0133] This disclosure provides a data annotation quality detection device, comprising:
[0134] The fourth acquisition module is used to obtain the target image corresponding to the error detection result through the above-mentioned data annotation quality detection method;
[0135] The image verification module is used to verify the actual annotation information of the target image corresponding to the error detection result.
[0136] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.
[0137] In this embodiment, the data labeling quality detection device and the data labeling quality verification device are presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.
[0138] This invention also provides a computer device having the above-described data annotation quality detection device and data annotation quality verification device.
[0139] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 6 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take a processor 10 as an example.
[0140] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.
[0141] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.
[0142] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0143] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.
[0144] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.
[0145] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.
[0146] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.
Claims
1. A method for detecting the quality of data annotation, characterized in that, The method includes: Obtain multiple target images and the current annotation results for each target image; Extract current annotation features from the current annotation results of each target image, wherein the current annotation features include correct annotation identifiers and incorrect annotation identifiers; The current labeled features extracted from each target image are input into the target detection model for detection, and the corresponding data quality detection results for each target image are detected. The target detection model is a pre-trained neural network, and the data quality detection results for each target image include: correct detection results, incorrect detection results, and missed detection results.
2. The method according to claim 1, characterized in that, Before the step of acquiring multiple target images, the following is also included: Obtain multiple historical images and the historical annotation results for each historical image; Historical annotation features are extracted from the historical annotation results of each historical image, and the historical annotation features include correct annotation markers and incorrect annotation markers; Historical annotation features are input into the target detection model for training, generating data quality detection results for each historical image, as well as the loss between the historical annotation results and the data quality detection results for each historical image. The loss is used to update the parameter values of the target detection model. The data quality detection results for each historical image include: correct detection results, incorrect detection results, and missed detection results.
3. The method according to claim 2, characterized in that, include: Use all of the aforementioned historical images as the test set; Obtain the average category accuracy for each historical image in the test set; If the average class accuracy of each historical image in the test set is greater than or equal to a preset threshold, then the class labeling quality of the multiple historical images is determined to meet the requirements. If the average category accuracy of each historical image in the test set is not greater than a preset threshold, then the category labeling quality of the multiple historical images is determined to be unsatisfactory.
4. The method according to claim 3, characterized in that, Obtain the average class accuracy for each historical image in the test set, including: Extract the current annotation features from the current annotation results of each historical image in the test set; The current annotation features extracted from the current annotation results of each historical image in the test set are input into the target detection model for training, and the corresponding predictions are made for the data quality detection results of each historical image in the test set and the loss between the historical annotation results of each historical image and the data quality detection results of each historical image. The average category accuracy of each historical image in the test set is calculated based on the loss between the data quality detection result of each historical image in the test set and the current annotation result of that historical image. Calculate the average class accuracy for each historical image in the test set based on the average class accuracy for each historical image in the test set.
5. The method according to claim 3, characterized in that, include: Based on the loss between the data quality detection result of each historical image in the test set and the current annotation result of that historical image, the recall and precision of each historical image in the test set are calculated.
6. The method according to any one of claims 2 to 4, characterized in that, include: Based on the loss between the historical annotation results and the data quality detection results of each historical image, the sum of the losses of the multiple historical images is calculated. Calculate the average of the sum of losses from the multiple historical images; The average of the sum of losses of the multiple historical images is used as the loss between the final historical annotation result of each historical image and the data quality detection result of that historical image.
7. A method for verifying the quality of data annotation, characterized in that, include: The target image corresponding to the error detection result is obtained by the data annotation quality detection method according to any one of claims 1 to 6; Verify the actual annotation information of the target image corresponding to the error detection result.
8. A device for detecting the quality of data annotation, characterized in that, The device includes: The acquisition module is used to acquire multiple target images and the current annotation results for each target image; An extraction module is used to extract current annotation features from the current annotation results of each target image, wherein the current annotation features include correct annotation identifiers and incorrect annotation identifiers; The detection module is used to input the current labeled features extracted from each target image into the target detection model for detection, and to detect the data quality detection results of each target image. The target detection model is a pre-trained neural network, and the data quality detection results of each target image include: correct detection results, incorrect detection results, and missed detection results.
9. A computer device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the data annotation quality detection method of any one of claims 1 to 6, or the data annotation quality verification method of claim 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the data annotation quality detection method according to any one of claims 1 to 6, or the data annotation quality verification method according to claim 7.