An image labeling quality inspection method and device, electronic equipment and medium

Abstract

This application discloses an image annotation quality inspection method, apparatus, electronic device, and medium. The method includes: if two personnel have the same image annotation accuracy, extracting a preset number of annotated target region images from the annotated images of both personnel and sending them to the other personnel's terminal to continue the image annotation verification task; if the accuracy rates are inconsistent, increasing the basic task for the personnel with lower image annotation accuracy and increasing the verification task for the personnel with higher image annotation accuracy; and determining the image annotation quality inspection result based on the results of the verification tasks performed by the two personnel. This scheme allows two personnel to verify the annotated target region images, thereby verifying the accuracy of the annotated target results and improving the accuracy of the target annotation results. Dynamically adjusting the number of target regions and the number of basic regions in the verification tasks for the two personnel achieves a balance in workload and improves work efficiency and the accuracy of image annotation quality inspection.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image annotation quality inspection method, apparatus, electronic device and medium. Background Technology

[0002] With the rapid development and product deployment of artificial intelligence (AI) technology, AI products are now ubiquitous in our lives, from facial recognition access control in residential communities and facial unlocking on mobile phones to automatic image capture and tracking by cameras and automatic identification by instruments. All these AI applications rely heavily on data. Data is the most fundamental of the three essential elements of AI (data, computing power, and algorithms) and a prerequisite for training large-scale AI models. The quality of data annotation is crucial, directly impacting the performance of AI models and the effectiveness of their applications. Generally, data annotation quality refers primarily to the accuracy of the annotations. Quality control of annotated data is a critical step in data annotation activities, directly affecting the accuracy of model training. For some high-quality data annotation projects, the workload and cost of quality control are extremely large.

[0003] Current data annotation quality control processes sometimes require manual, one-by-one inspection or manual sampling inspection. In some cases, errors by quality control personnel may lead to data quality failing to meet requirements. Furthermore, these processes are labor-intensive and time-consuming. Summary of the Invention

[0004] This application provides an image annotation quality inspection method, apparatus, electronic device, and medium to select a portion of the targets already annotated by personnel and send them to other personnel for cross-verification, and dynamically adjust the number of targets to be verified, thereby improving the accuracy of image annotation quality inspection and achieving a balance of workload among personnel.

[0005] According to one aspect of this application, an image annotation quality inspection method is provided, the method comprising:

[0006] If two people perform image annotation tasks on their respective images, and the two people have the same annotation accuracy, then a preset number of annotated target area images are extracted from the two people's annotated images and sent to the other person's terminal to continue the image annotation verification task.

[0007] If the image annotation accuracy rates of two personnel are inconsistent, more than a preset number of labeled target area images will be extracted from the labeled images of the personnel with lower image annotation accuracy and sent to the other personnel's terminal to continue the verification task.

[0008] The system extracts fewer than a preset number of labeled target region images from the labeled images of personnel with high image annotation accuracy and sends them to the other personnel's terminal to continue the verification task. It also selects target images to be labeled from the unlabeled images of personnel with high image annotation accuracy and sends them to the other personnel's terminal to continue the basic task. The image annotation accuracy is updated based on the quality inspection results of the image annotations of the two personnel.

[0009] Based on the results of the two personnel performing the verification task, the quality inspection results of the image annotations of the two personnel are determined.

[0010] According to one aspect of this application, an image annotation quality inspection device is provided, the device comprising:

[0011] The first sending module is used to extract a preset number of labeled target area images from the labeled images of the two personnel when the two personnel have the same image labeling accuracy, and send them to the other personnel's terminal to continue the image labeling verification task.

[0012] The second sending module is used to extract more than a preset number of labeled target area images from the labeled images of the person with the lower image labeling accuracy if the image labeling accuracy of the two personnel is inconsistent, and send them to the other person's terminal to continue the verification task.

[0013] The third sending module is used to extract fewer than a preset number of labeled target area images from the labeled images of personnel with high image annotation accuracy, and send them to the other personnel's terminal to continue the verification task; and to select target images to be labeled from the images to be labeled of personnel with high image annotation accuracy, and send them to the other personnel's terminal to continue the basic task; wherein, the image annotation accuracy is updated according to the quality inspection results of the image annotation of the two personnel;

[0014] An update module is used to determine the quality inspection results of image annotations by the two personnel based on the results of the verification task performed by the two personnel.

[0015] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory that is communicatively connected to at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the image annotation quality inspection method of any embodiment of this application.

[0019] According to another aspect of this application, a computer-readable storage medium is provided, which stores computer instructions for causing a processor to execute and implement the image annotation quality inspection method of any embodiment of this application.

[0020] The technical solution of this application embodiment, when two personnel are performing basic image annotation tasks on images to be annotated, if the image annotation accuracy rates of the two personnel are the same, a preset number of annotated target region images are extracted from the annotated images of the two personnel and sent to the other personnel's terminal to continue the image annotation verification task; if the image annotation accuracy rates of the two personnel are different, more than a preset number of annotated target region images are extracted from the annotated images of the personnel with lower image annotation accuracy and sent to the other personnel's terminal to continue the verification task; less than a preset number of annotated target region images are extracted from the annotated images of the personnel with higher image annotation accuracy and sent to the other personnel's terminal to continue the verification task; and a target image to be annotated is selected from the images to be annotated by the personnel with higher image annotation accuracy and sent to the other personnel's terminal to continue the basic task; based on the results of the two personnel performing the verification task, the quality inspection results of the image annotation of the two personnel are determined. The above solution can select annotated target region images and send them to the other personnel for image annotation verification again when two personnel are performing basic image annotation tasks, thereby realizing the verification of the accuracy of the annotated target results and improving the accuracy of the target results. In addition, during the quality inspection process, the number of target images for the verification tasks and the number of images for the basic tasks of the two personnel can be dynamically adjusted adaptively according to the changes in the accuracy of the annotation, thereby achieving a balance of workload. Furthermore, tasks can be adaptively allocated according to the characteristics of personnel image annotation, improving work efficiency and the accuracy of image annotation quality inspection.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart of an image annotation quality inspection method provided in this application embodiment;

[0024] Figure 2A flowchart of an image annotation quality inspection method is provided in another embodiment of this application;

[0025] Figure 3 This is a schematic diagram of the structure of an image annotation quality inspection device provided in an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," "third," "fourth," "actual," "preset," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Figure 1 This is a flowchart illustrating an image annotation quality inspection method provided in an embodiment of this application. This embodiment is applicable to situations requiring verification of image annotation results performed by personnel. The method can be executed by an image annotation quality inspection device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0030] S110. In the case where two people are performing basic image annotation tasks on images to be annotated, if the image annotation accuracy of the two people is the same, then a preset number of annotated target area images are extracted from the annotated images of the two people and sent to the other person's terminal to continue the image annotation verification task.

[0031] In this cross-validation process, two people form a group, and multiple groups can exist. Grouping people into two groups for cross-validation ensures the accuracy of image annotation quality control and improves validation efficiency by avoiding duplicate validation. For example, consider personnel A, B, C, and D. 100 images to be labeled are assigned to these four personnel, with each person receiving 25 images. Personnel A and B can be grouped together to cross-validate their annotation results. Similarly, personnel C and D can be grouped together to cross-validate their annotation results.

[0032] An image to be labeled is an image that needs to be labeled but has not yet been labeled. A labeled image is an image that has already been labeled. The basic task is to label the image to be labeled, and the accuracy of the image labeling is the ratio of the number of correctly labeled targets to the total number of labeled targets. The preset number can be determined according to the actual situation, and can be less than or equal to the number of labeled targets. A labeled target area image is a region image in a labeled image that contains labeled targets. The other person is the other person among the two people besides the person who extracted the labeled target area image. For example, if a preset number of labeled target area images are extracted from person A's labeled image, then the other person is person B; if a preset number of labeled target area images are extracted from person B's labeled image, then the other person is person A. The image labeling verification task is to perform the image labeling task again on the labeled target area images of the other person's labeled targets. The terminal is a device that provides an operating system for the two people to perform the basic image labeling task and the image labeling verification task respectively. The terminal and the person can be in a one-to-one correspondence. The electronic device executing the embodiments of this application can communicate with each terminal to obtain data from each terminal and send data to each terminal.

[0033] In this embodiment, two personnel are assigned the basic task of image annotation. During the execution of the basic task, if the image to be annotated has been completed, it becomes an annotated image. For the annotated targets in the annotated image, the annotated target region image needs to be extracted and sent to the other personnel for image annotation verification. The other personnel then perform image annotation verification on the received annotated target region image. If the image annotation accuracy of the two personnel is consistent, their annotation levels are consistent. Therefore, they can send each other the same number of annotated target region images, achieving a balance between workload and image annotation verification accuracy. Specifically, a preset number of annotated targets can be selected from the annotated images, and the region image of each annotated target can be extracted as an annotated target region image, sent to the other personnel for cross-verification, and a second image annotation verification task can be performed. The consistent image annotation accuracy of the two personnel includes two cases: one is when neither personnel has initially performed the basic image annotation task or the image annotation verification task, and there is no specific information on the image annotation accuracy; in this case, the initial value can be set to 0. Another scenario is that the basic tasks of image annotation and the verification tasks of image annotation have been performed, and the image annotation accuracy is updated in real time. The image annotation accuracy has a certain value, and the image annotation accuracy values ​​are the same.

[0034] S120. If the image annotation accuracy rates of the two personnel are inconsistent, extract more than a preset number of labeled target area images from the labeled images of the personnel with lower image annotation accuracy rates and send them to the other personnel's terminal to continue the verification task.

[0035] For example, when two individuals have different image annotation accuracy rates, their annotation levels are inconsistent. Their respective task types can be adaptively adjusted based on their annotation characteristics to improve work efficiency and annotation accuracy. For instance, an individual with a lower image annotation accuracy rate is considered to have a relatively higher probability of making annotation errors during basic image annotation tasks; that is, the probability of mislabeling already labeled targets in an already labeled image is higher. Therefore, more labeled targets can be selected, and more images of labeled target regions can be sent to an individual with a higher image annotation accuracy rate for verification, thereby improving the accuracy of image annotation.

[0036] Specifically, the image annotation accuracy of the two personnel is updated and monitored in real time. If the image annotation accuracy of the two personnel is inconsistent, the workload of their basic tasks and verification tasks can be dynamically adjusted. More than a preset number of annotated target area images can be extracted from the annotated images of the personnel with low image annotation accuracy and sent to the terminal of the personnel with high image annotation accuracy, so that the personnel with high accuracy can verify more annotated target area images.

[0037] S130. Extract fewer than a preset number of labeled target region images from the labeled images of personnel with high image annotation accuracy, and send them to the other personnel's terminal to continue the verification task. Also, select target images to be labeled from the unlabeled images of personnel with high image annotation accuracy, and send them to the other personnel's terminal to continue the basic task. The image annotation accuracy is updated based on the quality inspection results of the image annotations from both personnel.

[0038] For example, if the image annotation accuracy of a person decreases, it can be assumed that the confidence of the person's verification task also decreases. Therefore, the verification task of the person with low image annotation accuracy can be adaptively reduced. Less than a preset number of labeled target area images can be extracted from the labeled images of the person with high image annotation accuracy and sent to the terminal of the person with low image annotation accuracy, so as to reduce the verification task of the person with low image annotation accuracy.

[0039] When the workload of verification tasks increases for personnel with high image annotation accuracy, the basic tasks for these personnel can be appropriately reduced and allocated to those with lower accuracy to achieve a balance in workload. Furthermore, assigning more basic tasks to personnel with lower accuracy to increase their workload can also increase their proficiency and attention to basic image annotation tasks, leading to a qualitative change through quantitative accumulation and indirectly improving their image annotation accuracy.

[0040] Specifically, in order to balance the workload, target images can be selected from the images to be labeled by personnel with high image labeling accuracy and sent to the terminals of personnel with low image labeling accuracy. This allows personnel with low image labeling accuracy to label more images, thereby improving their image labeling proficiency while achieving a balance in workload.

[0041] In this embodiment of the application, the image annotation accuracy is updated in real time based on the results of the verification task as two people perform the verification task.

[0042] S140. Based on the results of the two personnel performing the verification task, determine the quality inspection results of the image annotations of the two personnel.

[0043] For example, during the basic task of image annotation, two personnel, after extracting a specified number of labeled target region images from the existing labeled images, send them to each other for verification. After both personnel complete the verification task, they obtain the results, and the verification results of their image annotations are determined based on these results. For instance, assuming the other personnel's annotation of the target is consistent with the original personnel's annotation, the quality inspection of the original personnel's image annotation is deemed passed; otherwise, the quality inspection result is deemed failed, or the image is sent to a third-party quality inspector for further inspection. After determining the quality inspection results of the personnel's image annotations, the image annotation accuracy rates of both personnel are updated, which are used to determine the number of labeled target region images to be extracted again for verification in subsequent extractions.

[0044] The technical solution of this application embodiment, when two personnel are performing basic image annotation tasks on images to be annotated, if the image annotation accuracy rates of the two personnel are the same, a preset number of annotated target region images are extracted from the annotated images of the two personnel and sent to the other personnel's terminal to continue the image annotation verification task; if the image annotation accuracy rates of the two personnel are different, more than a preset number of annotated target region images are extracted from the annotated images of the personnel with lower image annotation accuracy and sent to the other personnel's terminal to continue the verification task, and target images to be annotated are selected from the images to be annotated by the personnel with higher image annotation accuracy and sent to the other personnel's terminal to continue the basic task; based on the results of the two personnel performing the verification task, the quality inspection results of the image annotation of the two personnel are determined. The above solution can select annotated target region images and send them to the other personnel for image annotation verification again when two personnel are performing basic image annotation tasks, thereby realizing the verification of the accuracy of the annotated target results and improving the accuracy of the target results. In addition, during the verification process, the number of target images for the two personnel's verification tasks and the number of images for the basic tasks are dynamically adjusted adaptively based on changes in the annotation accuracy, thereby achieving a balance of workload. Furthermore, tasks can be adaptively allocated according to the characteristics of personnel's image annotation, improving work efficiency and the accuracy of image annotation quality inspection.

[0045] The following embodiments can be applied to the processes of "extracting a preset number of labeled target region images from labeled images of two individuals", "extracting more than a preset number of labeled target region images from labeled images of individuals with low image annotation accuracy", and "extracting less than a preset number of labeled target region images from labeled images of individuals with high image annotation accuracy". All describe the extraction process of labeled target region images. If applied to "extracting a preset number of labeled target region images from labeled images of two individuals", the number to be verified is the preset number. If applied to "extracting more than a preset number of labeled target region images from labeled images of individuals with low image annotation accuracy", the number to be verified is the number exceeding the preset number. If applied to "extracting less than a preset number of labeled target region images from labeled images of individuals with high image annotation accuracy", the number to be verified is the number below the preset number.

[0046] As a non-limiting implementation, the method involves extracting a number of labeled target region images to be verified from labeled images, including:

[0047] For each labeled target in the labeled image, the labeling difficulty score of the labeled target is determined based on at least one of the following: the labeling time of the user for the labeled target, the number of times the labeling result is modified, and the time spent modifying the labeling result.

[0048] Based on the annotation difficulty score, extract the labeled target region images corresponding to the number of most difficult labeled targets to be verified;

[0049] The quantity to be verified includes a preset quantity, a quantity exceeding the preset quantity, and a quantity below the preset quantity.

[0050] For example, for each labeled target in an already labeled image, during the process of a person annotating the image on the terminal display interface, the terminal can detect at least one of the following: annotation duration, number of times the annotation result is modified, and time spent modifying the annotation result. Annotation duration can be the time from when the terminal detects that a person begins annotating a target in the image until the person clicks to complete the annotation of that target. Number of times the annotation result is modified is the number of times the user modifies the completed annotation result; modifications can include moving, deleting, scaling the detection box drawn by the user for the labeled target, and modifying label attributes. Time spent modifying the annotation result is the total time required for the entire modification process. At least one of the annotation duration, number of times the annotation result is modified, and time spent modifying the annotation result can be positively correlated with the annotation difficulty score; that is, the higher the annotation difficulty score, the greater the annotation difficulty. At least one of the annotation duration, number of times the annotation result is modified, and time spent modifying the annotation result can be standardized or normalized to bring them to the same dimension before further calculating the annotation difficulty score. One possible approach is to determine the annotation difficulty score based on the annotation duration, number of annotation result modifications, and time taken to modify the annotation result of the labeled targets. Then, the labeled targets are sorted by annotation duration in ascending order, and the sorted number is used as the annotation duration score. Next, they are sorted by the number of annotation result modifications in ascending order, and the sorted number is used as the annotation result modification count score. Finally, they are sorted by the time taken to modify the annotation result in ascending order, and the sorted number is used as the annotation result time score. The annotation difficulty score is then obtained by weighted summing of these scores. For example, it can be represented as `time`. n *0.3+resize n *0.3+changeTime n *0.4, time n The annotation duration score is 0.3, which is the weight corresponding to the annotation duration score. (resize) n The score represents the number of times the annotation result was modified, with 0.3 being the weight corresponding to the number of times the annotation result was modified, and changeTime. n The score represents the time spent modifying the annotation results, with 0.4 being the weight corresponding to the time spent modifying the annotation results. The weight value can also be adjusted adaptively.

[0051] For example, the more difficult the labeled target is to annotate, the greater the probability of an incorrect standard result. Therefore, it should be selected as the labeled target and sent to the other party for verification. Based on the annotation difficulty score, a number of the most difficult labeled targets to be verified are selected. The labeled target region images corresponding to these most difficult labeled targets are extracted from the labeled image and sent to the other party for image annotation verification. Specifically, if the annotation difficulty score and annotation difficulty are positively correlated, a number of labeled targets with the highest annotation difficulty score can be selected. The labeled target region images corresponding to these highest-scoring labeled targets are extracted and sent to the other party for verification.

[0052] As a non-limiting implementation, the method involves extracting a number of labeled target region images to be verified from labeled images, including:

[0053] The personnel are responsible for drawing the detection bounding boxes for the selected labeled targets;

[0054] Based on the detection frame, a preset pixel range is expanded outward to obtain an image of the labeled target area; wherein, the number to be verified includes a preset number, a number exceeding the preset number, and a number below the preset number.

[0055] For example, in the process of personnel annotating targets in an already labeled image, a detection box is drawn to reflect the range and position of the labeled targets. The terminal can detect the detection box drawn by the personnel. A preset pixel range can be extended outward based on the detection box to obtain an image of the labeled target area. The preset pixel range can be determined according to the actual situation, for example, it can be set to 10 pixels. Assuming that the detection box is a rectangular detection box, the top edge, bottom edge, left edge, and right edge of the detection box are each extended outward by 10 pixels to obtain an image of the labeled target area. The beneficial effect is that the detection box annotated by personnel during the basic task may not be accurate, and may be offset or scaled relative to the actual range of the labeled targets. If the image of the area within the detection box range is directly extracted and sent to the other party for verification, the incompleteness of the labeled targets in the detection box may lead to labeling errors during the verification process. In this embodiment of the application, the preset pixel range is expanded on the basis of the detection box, which can include the labeled targets as completely as possible, so that the image of the labeled target area can contain the complete labeled targets, making it easier for the other party to more accurately annotate the range and position of the labeled targets, and improving the accuracy of image annotation. In addition, the labeled target area images in the verification task are all small target images dynamically selected based on the labeling difficulty. When the other party is performing verification, it is easy to realize that they are labeling verification targets. Moreover, the small images have less interference, so the accuracy of the labeled targets will be higher, making them more accurate as comparison verification items, and reducing the intervention of third-party quality inspection.

[0056] As a non-limiting implementation, when the number to be verified includes a preset number, a number exceeding the preset number, and a number below the preset number, and when the two personnel include a person with high image annotation accuracy and a person with low image annotation accuracy, before extracting the labeled target region images of the number to be verified from the labeled images of the two personnel, the method further includes:

[0057] Based on the total number of images to be labeled and the number of personnel, determine the number of images to be labeled that each personnel needs to be assigned.

[0058] Based on the preset number of verification targets, the number of images to be labeled, and the image labeling accuracy of the two personnel, the number of images to be verified is determined, and the required number of labeled images needed to extract the labeled target area images of the number of images to be verified is determined.

[0059] Accordingly, extract the number of labeled target region images to be verified from the labeled images, including:

[0060] During the basic task of personnel traversing the images to be labeled and performing image labeling, if the number of newly labeled images reaches the required number, then the number of labeled target area images to be verified is extracted from the required number of labeled images.

[0061] In an embodiment of the present application, when the quantity to be verified includes a preset quantity, a quantity exceeding the preset quantity, and a quantity less than the preset quantity, and the two personnel include a person with a high image annotation accuracy rate and a person with a low image annotation accuracy rate, extracting the annotated target region images of the quantity to be verified from the annotated images of the two personnel includes extracting the preset quantity of annotated target region images from the annotated images of the two personnel, extracting the annotated target region images exceeding the preset quantity from the annotated images of the person with a low image annotation accuracy rate, and extracting the annotated target region images less than the preset quantity from the annotated images of the person with a high image annotation accuracy rate. Here, the quantity of images to be annotated is the quantity of images to be annotated for each person. The preset verification target number is the upper limit of the quantity of annotated target region images that need to be verified in the verification task for each person. Exemplarily, the preset verification target number can be set in advance to control the task volume of each person performing the verification task, avoid personnel repeatedly performing a large number of verification tasks, resulting in excessive ineffective verification, and affecting the efficiency of the basic task of image annotation. The quantity of annotated target region images that need to be extracted after the annotation of each batch of images to be annotated can be reflected according to the preset verification target number, the quantity of images to be annotated, and the image annotation accuracy rates of the two personnel, that is, the quantity to be verified, and further determine the required quantity of annotated images for extracting the annotated target region images of the quantity to be verified. For example, assume that the preset verification target number is P, and the quantity of images to be annotated is N, then the preset quantity Y = P / N. If the image annotation accuracy rates of the two personnel are the same, then the preset quantity Y is used as the quantity to be verified Q n . If the image annotation accuracy rates of the two personnel are different, then it is necessary to extract the annotated target region images exceeding the preset quantity Y from the annotated images of the person with a low image annotation accuracy rate. For the person with a low image annotation accuracy rate Q n >Y, extract the annotated target region images less than the preset quantity Y from the annotated images of the person with a high image annotation accuracy rate. For the person with a high image annotation accuracy rate Q n <Y. If Q n >1, it reflects that Q n annotated target region images need to be extracted from each annotated image. At this time, the required quantity is 1, that is, for each annotated image obtained by the personnel, Q n annotated target region images are extracted from the annotated image. If Q n <1, it reflects that one annotated target region image needs to be extracted from multiple annotated images. At this time, the required quantity is 1 / Q n , that is, for every 1 / Q n annotated images obtained by the personnel, one annotated target region image is extracted from 1 / Q n annotated images.

[0062] When personnel traverse the images to be labeled to perform basic tasks, after labeling the required number of images to be labeled and obtaining the required number of labeled images, the required number of labeled target region images to be verified are extracted from the labeled images and sent to the other personnel for cross-verification. This improves the real-time performance of cross-verification, allows for the timely detection and correction of labeling issues, and enables timely updates to the image labeling accuracy of both personnel based on real-time quality inspection results. This facilitates dynamic adjustment of the number of verification tasks and basic tasks for both personnel. For example, suppose Q... n If the value is greater than 1, then for each image to be labeled by two people, a labeled image is obtained. Q is then extracted from the labeled image. n An image of the labeled target area is sent to the other party for verification. Based on the verification result, the quality inspection result of the image labeling is determined. The image labeling accuracy of the two personnel is updated in real time based on the quality inspection result. The Q is adjusted in real time based on the updated image labeling accuracy. n Make adaptive dynamic adjustments.

[0063] Figure 2 This is a flowchart of an image annotation quality inspection method provided in another embodiment of this application. This embodiment is an optimization based on the above embodiment; solutions not described in detail in this embodiment are found in the above embodiment. Figure 2 As shown, the method in this embodiment of the application specifically includes the following steps:

[0064] S210. The product of the pre-set verification target offset and the absolute value of the difference in image annotation accuracy between the two personnel is used as the adjustment quantity.

[0065] The pre-set verification target offset reflects the number of labeled targets in the verification task that needs adjustment when the image annotation accuracy of two personnel is inconsistent. The adjustment quantity, i.e., the number of labeled targets in the verification task that needs adjustment, can be the product of the verification target offset and the absolute value of the difference in image annotation accuracy between the two personnel. Assuming the verification target offset is S, one person's image annotation accuracy is E1, and the other person's is E2, then the adjustment quantity can be S*|E1-E2|. The adjustment quantity can be limited to ensure that adding the adjustment quantity to the preset quantity results in a number less than or equal to the preset number of verification targets. The adjustment quantity can be greater than 0, less than or equal to the difference between the preset number of verification targets and the preset quantity, further limiting the verification target offset S. This scheme adaptively adjusts the adjustment quantity based on the image annotation accuracy of the two personnel, allowing the adjustment quantity to dynamically change to meet the image annotation verification requirements under the current situation. Different image annotation accuracy rates result in different degrees of verification task skewness, thus improving image annotation efficiency and accuracy through dynamic adjustment.

[0066] S220. The sum of the preset quantity and the adjusted quantity is taken as the first adjustment verification target quantity.

[0067] For example, the sum of the preset quantity and the adjusted quantity is used as the first adjustment verification target quantity, i.e., Q1 = Y + S, where Q1 is the first adjustment verification target quantity, reflecting the quantity exceeding the preset quantity.

[0068] S230. The difference between the preset quantity and the adjusted quantity is taken as the second adjustment verification target quantity.

[0069] For example, the difference between the preset quantity and the adjusted quantity is used as the second adjustment verification target quantity, i.e., Q2 = YS, where Q2 is the second adjustment verification target quantity, reflecting the quantity that is lower than the preset quantity.

[0070] S240. In the case where two people are performing basic image annotation tasks on images to be annotated, if the image annotation accuracy of the two people is the same, then a preset number of annotated target area images are extracted from the annotated images of the two people and sent to the other person's terminal to continue the image annotation verification task.

[0071] S250. If the total number of labeled target area images performed by personnel with high image labeling accuracy does not exceed the preset number of verification targets, select labeled target area images with the first adjusted number of verification targets from the labeled images of personnel with low image labeling accuracy and send them to the other personnel's terminal to continue the verification task.

[0072] For example, for personnel with high image annotation accuracy, the number of labeled target region images they perform verification tasks needs to be increased. In other words, the number of labeled target region images verified by personnel with high image annotation accuracy is increased. However, this increased number needs to be limited to avoid personnel repeatedly performing a large number of verification tasks, leading to excessive invalid verification and affecting the efficiency of their basic image annotation tasks. Therefore, it is necessary to adaptively increase the verification task volume while ensuring that the total number of labeled target region images performed by personnel with high image annotation accuracy does not exceed the preset verification target number.

[0073] Specifically, from the labeled images of personnel with low image annotation accuracy, a first number of labeled target area images for verification are selected, thereby extracting more than a preset number of labeled target area images and sending them to the terminal of personnel with high image standard accuracy, so that personnel with high image annotation accuracy can perform verification and improve the accuracy of verification.

[0074] S260. Extract the labeled target area image of the second adjusted verification target from the labeled image of the personnel with high image labeling accuracy, and send it to the terminal of the other personnel to continue the verification task.

[0075] For example, the confidence level of the verification results performed by personnel with low image annotation accuracy is reduced. Their verification workload can be adaptively reduced by extracting the second set of labeled target area images with adjusted verification target quantity from the labeled images of personnel with low image annotation accuracy and sending them to the terminal of personnel with low image annotation accuracy so that they can perform verification.

[0076] S270. Select the adjusted number of target images to be labeled from the images to be labeled by personnel with high image labeling accuracy, and send them to the other personnel's terminal to continue the basic task.

[0077] For example, when the verification workload of personnel with high image annotation accuracy is increased, in order to balance the workload of personnel with high and low image annotation accuracy, the basic workload of personnel with low image annotation accuracy can be adaptively increased. An adjusted number of target images to be annotated are selected from the images to be annotated by personnel with high image annotation accuracy and sent to the terminal of personnel with low image annotation accuracy to continue to perform basic tasks, thereby balancing the workload of the two personnel to a certain extent.

[0078] It should be noted that personnel with high image annotation accuracy increased the number of already labeled target area images for verification tasks and reduced the number of unlabeled images for basic tasks. Conversely, personnel with low image annotation accuracy increased the number of unlabeled images for basic tasks and reduced the number of already labeled target area images for verification tasks. However, each unlabeled image may contain at least one unlabeled target. Therefore, the number of unlabeled targets added by personnel with low accuracy may be greater than the adjusted number, but overall, the workload is roughly balanced. Alternatively, fewer unlabeled target images than the adjusted number can be selected from the unlabeled images of personnel with high accuracy and sent to the terminal of personnel with low accuracy to continue basic tasks, thus balancing the number of unlabeled targets added by the two personnel.

[0079] S280. Based on the results of the two personnel performing the verification task, determine the quality inspection results of the image annotations of the two personnel.

[0080] In this embodiment of the application, the process of determining the result of the verification task includes:

[0081] Obtain the annotation attributes and annotation locations of two individuals for the same labeled target;

[0082] If two people have inconsistent annotation attributes for the same labeled target, the result of the verification task is determined to be unsuccessful.

[0083] If two people have the same annotation attributes for the same labeled target, then the similarity of the labeled locations is determined.

[0084] If the similarity exceeds the similarity threshold, the result of the verification task is determined to be passed; otherwise, the result of the verification task is determined to be failed.

[0085] For example, the annotation results of personnel for labeled targets can include annotation attributes and annotation locations. Annotation attributes include feature information such as the classification, color, and shape of the labeled target, while annotation locations can include the range of the labeled target represented by a detection box and its position relative to the labeled image. Generally, the severity of annotation attribute errors is higher than that of annotation location errors. Therefore, the annotation attributes of two personnel for the same labeled target can be compared first. If they are inconsistent, the result of the verification task is directly determined to be a failure. If the annotation attributes of two personnel groups for the same labeled target are consistent, the annotation locations are further compared. Specifically, the similarity between the two annotation locations is calculated. If the similarity exceeds the similarity threshold, the result of the verification task is determined to be a pass; if the similarity does not exceed the similarity threshold, the result of the verification task is determined to be a failure. The similarity between two annotation locations can be calculated as follows: Assume that the annotation locations of personnel A for the same labeled target include N coordinates: (a1, b1), (a2, b2)...(a...b1)...(a2, b2 ... n ,b n Personnel B has N coordinates for the location of the marked target: (a′1, b′1), (a′2, b′2)...(a′...b′1). n ,b′ n If the similarity between the two labeled locations is , then the similarity between the two labeled locations is . Other methods can also be used to calculate similarity; there are no restrictions. If the similarity θ exceeds the similarity threshold SI, the verification task is considered passed, meaning the quality control of the person's image annotation is satisfactory. If the similarity θ does not exceed the similarity threshold SI, the verification task is considered failed.

[0086] If the verification task fails, the annotation results of both personnel are sent to the terminal of a third-party quality inspector. The third-party quality inspector conducts further quality inspection to determine the final quality inspection result and updates the image annotation accuracy rates of both personnel based on the final quality inspection result. For example, if the third-party quality inspector determines that personnel A's annotation is correct and personnel B's annotation is incorrect, personnel B's image annotation accuracy rate will decrease. If personnel A's image annotation accuracy rate is 100%, its accuracy rate remains unchanged; if it is below 100%, it increases. If, after updating the image annotation accuracy rates based on the third-party quality inspector's final inspection result, a personnel's image annotation accuracy rate falls below a preset accuracy threshold, an alert is issued, prompting that personnel to re-annotate the already annotated images. The re-annotated results are then compared with the annotation results of another personnel to determine if the re-annotation is correct.

[0087] This application provides an image annotation quality inspection method. The method uses the product of a pre-set verification target offset and the absolute value of the difference in image annotation accuracy between two personnel as an adjustment quantity. The sum of the preset quantity and the adjustment quantity is used as the first adjusted verification target quantity. The difference between the preset quantity and the adjustment quantity is used as the second adjusted verification target quantity. If the total number of labeled target region images performed by the personnel with high image annotation accuracy does not exceed the preset verification target quantity, the method selects labeled target region images of the first adjusted verification target quantity from the labeled images of the personnel with low image annotation accuracy, extracts labeled target region images of the second adjusted verification target quantity from the labeled images of the personnel with high image annotation accuracy, and selects target images of the adjusted quantity from the images to be annotated by the personnel with high image annotation accuracy. This method can adaptively and dynamically adjust the task preferences of the two personnel based on their image annotation accuracy, fully utilizing their operational characteristics to allocate tasks, thereby improving image annotation efficiency and accuracy.

[0088] Figure 3 This is a schematic diagram of an image annotation quality inspection device provided in an embodiment of this application. This device can execute the image annotation quality inspection method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method. Figure 3 As shown, the device includes:

[0089] The first sending module 310 is used to extract a preset number of labeled target area images from the labeled images of the two personnel when the two personnel have the same image labeling accuracy, and send them to the other personnel's terminal to continue the image labeling verification task.

[0090] The second sending module 320 is used to extract more than a preset number of labeled target area images from the labeled images of the person with the lower image labeling accuracy if the image labeling accuracy of the two personnel is inconsistent, and send them to the other person's terminal to continue the verification task.

[0091] The third sending module 330 is used to extract fewer than a preset number of labeled target region images from the labeled images of personnel with high image annotation accuracy, and send them to the terminal of the other personnel to continue the verification task; and to select target images to be labeled from the unlabeled images of personnel with high image annotation accuracy, and send them to the terminal of the other personnel to continue the basic task. The image annotation accuracy is updated based on the quality inspection results of the image annotations of the two personnel.

[0092] The update module 340 is used to determine the quality inspection results of the image annotations of the two personnel based on the results of the two personnel performing the verification task.

[0093] In this embodiment of the application, the apparatus further includes: an extraction module, configured to extract a number of labeled target region images to be verified from the labeled images, including:

[0094] For each labeled target in the labeled image, the labeling difficulty score of the labeled target is determined based on at least one of the following: the labeling time of the user for the labeled target, the number of times the labeling result is modified, and the time spent modifying the labeling result.

[0095] Based on the annotation difficulty score, extract the labeled target region images corresponding to the number of most difficult labeled targets to be verified;

[0096] The quantity to be verified includes a preset quantity, a quantity exceeding the preset quantity, and a quantity below the preset quantity.

[0097] In this embodiment of the application, the extraction module extracts the number of labeled target region images to be verified from the labeled images, including:

[0098] The personnel are responsible for drawing the detection bounding boxes for the selected labeled targets;

[0099] Based on the detection frame, a preset pixel range is expanded outward to obtain an image of the labeled target area; wherein, the number to be verified includes a preset number, a number exceeding the preset number, and a number below the preset number.

[0100] In this embodiment of the application, when the quantity to be verified includes a preset quantity, a quantity exceeding the preset quantity, and a quantity lower than the preset quantity, and the two personnel include a person with high image annotation accuracy and a person with low image annotation accuracy, before the extraction module extracts the labeled target region images of the quantity to be verified from the labeled images of the two personnel, the method further includes:

[0101] Based on the total number of images to be labeled and the number of personnel, determine the number of images to be labeled that each personnel needs to be assigned.

[0102] Based on the preset number of verification targets, the number of images to be labeled, and the image labeling accuracy of the two personnel, the number of images to be verified is determined, and the required number of labeled images needed to extract the labeled target area images of the number of images to be verified is determined.

[0103] Accordingly, extract the number of labeled target region images to be verified from the labeled images, including:

[0104] During the basic task of personnel traversing the images to be labeled and performing image labeling, if the number of newly labeled images reaches the required number, then the number of labeled target area images to be verified is extracted from the required number of labeled images.

[0105] In this embodiment of the application, the square device further includes:

[0106] The adjustment quantity determination module is used to take the product of the pre-set verification target offset and the absolute value of the difference in image annotation accuracy between the two personnel as the adjustment quantity;

[0107] The first adjustment verification target quantity determination module is used to take the sum of the preset quantity and the adjustment quantity as the first adjustment verification target quantity;

[0108] The second adjustment verification target quantity module is used to take the difference between the preset quantity and the adjusted quantity as the second adjustment verification target quantity.

[0109] In this embodiment of the application, the second sending module 320 extracts more than a preset number of labeled target region images from labeled images of personnel with low image annotation accuracy, including:

[0110] If the total number of labeled target area images performed by personnel with high image annotation accuracy does not exceed the preset number of verification targets, then a first set of labeled target area images with adjusted verification target counts is selected from the labeled images of personnel with low image annotation accuracy.

[0111] The third sending module 330 extracts fewer than a preset number of labeled target region images from the labeled images of personnel with high image annotation accuracy, including:

[0112] Extract the labeled target region images from the labeled images of personnel with high image annotation accuracy; and adjust the number of verified targets in the second set of images.

[0113] The third sending module 330 selects the target image to be labeled from the images to be labeled by personnel with high image labeling accuracy, including:

[0114] Select the adjusted number of target images to be labeled from the images to be labeled by personnel with high image labeling accuracy.

[0115] In this embodiment of the application, the device further includes:

[0116] The annotation result acquisition module is used to obtain the annotation attributes and annotation positions of two people for the same labeled target;

[0117] The annotation attribute comparison module is used to determine the result of the verification task as failing if two people have inconsistent annotation attributes for the same annotated target.

[0118] The similarity determination module is used to determine the similarity of the labeled locations if two personnel have the same labeling attributes for the same labeled target.

[0119] The similarity comparison module is used to determine the result of the verification task as passed if the similarity exceeds the similarity threshold, and otherwise determine the result of the verification task as failed.

[0120] The image annotation quality inspection device provided in this application embodiment can execute the image annotation quality inspection method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the method execution.

[0121] Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0122] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0123] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0124] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as image annotation quality inspection methods.

[0125] In some embodiments, the image annotation quality control method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the image annotation quality control method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the image annotation quality control method by any other suitable means (e.g., by means of firmware).

[0126] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0127] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable image annotation and quality control device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0129] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0130] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0131] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0132] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired information of the technical solution of this application can be achieved, and this is not limited herein.

[0133] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image annotation quality inspection method, characterized in that, The method includes: If two people perform image annotation tasks on their respective images, and the two people have the same annotation accuracy, then a preset number of annotated target area images are extracted from the two people's annotated images and sent to the other person's terminal to continue the image annotation verification task. If the image annotation accuracy rates of two personnel are inconsistent, more than a preset number of labeled target area images will be extracted from the labeled images of the personnel with lower image annotation accuracy and sent to the other personnel's terminal to continue the verification task. The system extracts fewer than a preset number of labeled target region images from the labeled images of personnel with high image annotation accuracy and sends them to the other personnel's terminal to continue the verification task. It also selects target images to be labeled from the unlabeled images of personnel with high image annotation accuracy and sends them to the other personnel's terminal to continue the basic task. The image annotation accuracy is updated based on the quality inspection results of the image annotations of the two personnel. Based on the results of the two personnel performing the verification task, the quality inspection results of the image annotations of the two personnel are determined.

2. The method according to claim 1, characterized in that, Extract the number of labeled target region images to be verified from the labeled images, including: For each labeled target in the labeled image, the labeling difficulty score of the labeled target is determined based on at least one of the following: the labeling time of the user for the labeled target, the number of times the labeling result is modified, and the time spent modifying the labeling result. Based on the annotation difficulty score, extract the labeled target region images corresponding to the number of difficult-to-label labeled targets to be verified; The quantity to be verified includes a preset quantity, a quantity exceeding the preset quantity, and a quantity below the preset quantity.

3. The method according to claim 1 or 2, characterized in that, Extract the number of labeled target region images to be verified from the labeled images of the two individuals, including: The personnel are responsible for drawing the detection bounding boxes for the selected labeled targets; Based on the detection frame, a preset pixel range is expanded outward to obtain an image of the labeled target area; wherein, the number to be verified includes a preset number, a number exceeding the preset number, and a number below the preset number.

4. The method according to claim 1 or 2, characterized in that, When the quantity to be verified includes a preset quantity, a quantity exceeding the preset quantity, and a quantity below the preset quantity, and when the two personnel include a person with high image annotation accuracy and a person with low image annotation accuracy, before extracting the labeled target region images of the quantity to be verified from the labeled images of the two personnel, the method further includes: Based on the total number of images to be labeled and the number of personnel, determine the number of images to be labeled that each personnel needs to be assigned. Based on the preset number of verification targets, the number of images to be labeled, and the image labeling accuracy of the two personnel, the number of images to be verified is determined, and the required number of labeled images needed to extract the labeled target area images of the number of images to be verified is determined. Accordingly, the number of labeled target region images to be verified is extracted from the labeled images of the two individuals, including: During the basic task of personnel traversing the images to be labeled and performing image labeling, if the number of newly labeled images reaches the required number, then the number of labeled target area images to be verified is extracted from the required number of labeled images.

5. The method according to claim 1, characterized in that, The method further includes: The product of the pre-set verification target offset and the absolute value of the difference in image annotation accuracy between the two personnel is used as the adjustment amount; The sum of the preset quantity and the adjusted quantity is taken as the first adjustment verification target quantity; The difference between the preset quantity and the adjusted quantity is used as the second adjustment verification target quantity.

6. The method according to claim 5, characterized in that, Extract more than a preset number of labeled target region images from labeled images of personnel with low image annotation accuracy, including: If the total number of labeled target area images performed by personnel with high image annotation accuracy does not exceed the preset number of verification targets, then a first set of labeled target area images with adjusted verification target counts is selected from the labeled images of personnel with low image annotation accuracy. Extracting fewer than a preset number of labeled target region images from images labeled by personnel with high image annotation accuracy, including: Extract the labeled target region images from the labeled images of personnel with high image annotation accuracy; and adjust the number of verified targets in the second set of images. Select target images for annotation from images to be annotated by personnel with high image annotation accuracy, including: Select the adjusted number of target images to be labeled from the images to be labeled by personnel with high image labeling accuracy.

7. The method according to claim 1, characterized in that, The process of determining the results of the verification task includes: Obtain the annotation attributes and annotation locations of two individuals for the same labeled target; If two people have inconsistent annotation attributes for the same labeled target, the result of the verification task is determined to be unsuccessful. If two people have the same annotation attributes for the same labeled target, then the similarity of the labeled locations is determined. If the similarity exceeds the similarity threshold, the result of the verification task is determined to be passed; otherwise, the result of the verification task is determined to be failed.

8. An image annotation quality inspection device, characterized in that, The device includes: The first sending module is used to extract a preset number of labeled target area images from the labeled images of the two personnel when the two personnel have the same image labeling accuracy, and send them to the other personnel's terminal to continue the image labeling verification task. The second sending module is used to extract more than a preset number of labeled target area images from the labeled images of the person with the lower image labeling accuracy if the image labeling accuracy of the two personnel is inconsistent, and send them to the other person's terminal to continue the verification task. The third sending module is used to extract fewer than a preset number of labeled target area images from the labeled images of personnel with high image annotation accuracy, and send them to the other personnel's terminal to continue the verification task; and to select target images to be labeled from the images to be labeled of personnel with high image annotation accuracy, and send them to the other personnel's terminal to continue the basic task; wherein, the image annotation accuracy is updated according to the quality inspection results of the image annotation of the two personnel; An update module is used to determine the quality inspection results of image annotations by the two personnel based on the results of the verification task performed by the two personnel.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the image annotation quality inspection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the image annotation quality inspection method according to any one of claims 1-7.

Images