A data annotation method and electronic device based on multi-model fusion

Abstract

This application relates to the field of image processing technology, and in particular to a data annotation method and electronic device based on multi-model fusion, which is used to solve the problem of low annotation speed in conventional annotation methods. The method is as follows: using multiple detection models associated with the annotation task, target detection is performed on each of the multiple images associated with the annotation task, and multiple detection results are obtained for each image; each detection result corresponds to a detection model; the detection results include at least one detection information corresponding to each object detected in the corresponding image; for each image, the following operations are performed: for an image, based on the multiple detection results corresponding to the image and a first confidence threshold, candidate detection information corresponding to each object in the image is selected; based on the candidate detection information corresponding to each object in the image, the annotation result corresponding to the image is obtained; thereby improving annotation efficiency, annotation speed, and accuracy.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a data annotation method and electronic device based on multi-model fusion. Background Technology

[0002] In the field of machine vision, massive amounts of collected images are used to train various models. Typically, before training these models, image annotation is required on these massive images. Currently, conventional image annotation work is mostly done manually, which is inefficient, slow, time-consuming, and costly. Summary of the Invention

[0003] This application provides a data annotation method and electronic device based on multi-model fusion, which can improve annotation efficiency, increase annotation speed, shorten annotation time, and reduce annotation cost.

[0004] The specific technical solutions provided in this application are as follows:

[0005] In a first aspect, embodiments of this application provide a data annotation method based on multi-model fusion, including:

[0006] Based on the annotation instructions for the annotation task, obtain multiple images associated with the annotation task;

[0007] Using multiple detection models associated with the annotation task, target detection is performed on each of the multiple images to obtain multiple detection results corresponding to each image; wherein, each detection result corresponds to a detection model; the detection result includes at least one detection information corresponding to each object detected in the corresponding image; each detection information corresponds to a confidence level;

[0008] For each of the images, perform the following operations:

[0009] For an image, based on multiple detection results corresponding to the image and a first confidence threshold, candidate detection information corresponding to each object in the image is selected.

[0010] Based on the candidate detection information corresponding to each object in the image, the annotation result corresponding to the image is obtained, wherein the annotation result includes the annotation information corresponding to each object in the image.

[0011] The method provided in this application embodiment performs target detection on multiple images associated with the annotation task based on each detection model to obtain detection results. Then, through a multi-model fusion decision-making method, the detection information in the multiple detection results corresponding to each image is filtered to select at least one candidate detection information of the real object in each image. Then, the multiple candidate detection information after filtering are fused to obtain the annotation result corresponding to each image. In this way, the precision and recall of the annotation result corresponding to each image are better than the prediction result of a single model. Annotators only need to perform secondary annotation based on this annotation result, and the annotation speed is better than the annotation speed based on single model prediction. At the same time, since multiple models are used to obtain the preliminary annotation result corresponding to each image, the annotation efficiency and speed are improved, the annotation time is shortened, and the annotation cost is reduced.

[0012] In one possible implementation, the step of filtering candidate detection information for each object in the image based on multiple detection results corresponding to the image and a first confidence threshold includes:

[0013] For each of the multiple detection results, perform the following operations:

[0014] For a detection result, determine the baseline detection information in the detection result, wherein the baseline detection information is the unprocessed detection information in the detection result;

[0015] For each determined benchmark detection information, perform the following operations:

[0016] For a benchmark detection information, based on the intersection-union ratio (IOU) of the benchmark detection information and each target detection information in other detection results, the matching detection information corresponding to the benchmark detection information in each other detection result is determined, wherein the target detection information is the detection information that has the same detection category as the benchmark detection information but is not matched.

[0017] Based on the confidence of the benchmark detection information and the confidence of the candidate detection information that is greater than the IOU threshold among all matching detection information, the reference confidence of the object corresponding to the benchmark detection information is determined.

[0018] When the reference confidence level is determined to be greater than the first confidence threshold, each candidate detection information that is greater than the IOU threshold is determined as the candidate detection information of the object corresponding to the benchmark detection information in the image.

[0019] In one possible implementation, determining the matching detection information corresponding to the benchmark detection information in each of the other detection results based on the intersection-union ratio (IOU) value of the benchmark detection information and each target detection information in the other detection results includes:

[0020] Determine the IOU value of the benchmark detection information with each target detection information in other detection results;

[0021] The other detection information corresponding to the maximum IOU value in each of the other detection results is used as the matching detection information;

[0022] The step of determining the reference confidence level of the object corresponding to the benchmark detection information based on the confidence level of the benchmark detection information and the candidate detection information greater than the IOU threshold among all matched detection information includes:

[0023] Based on the IOU value between all matching detection information and the benchmark detection information, and the IOU value between every two matching detection information in all matching detection information, each candidate detection information with an IOU value greater than the threshold is selected from all matching detection information and the benchmark detection information.

[0024] The highest confidence level among the confidence levels corresponding to each candidate detection information is determined as the reference confidence level of the object corresponding to the benchmark detection information.

[0025] In one possible implementation, the first confidence threshold includes multiple sub-thresholds;

[0026] If the number of identified candidate detection information is equal to the total number of the plurality of detection models, then the first sub-threshold among the plurality of sub-thresholds is taken as the first confidence threshold.

[0027] If the number is less than the total number but greater than a preset value, then the second sub-threshold among the multiple sub-thresholds is used as the first confidence threshold.

[0028] If the number is equal to the preset value, then the third sub-threshold among the multiple sub-thresholds is taken as the first confidence threshold;

[0029] Wherein, the first sub-threshold is less than or equal to the second sub-threshold, and the second sub-threshold is less than the third sub-threshold.

[0030] In one possible implementation, if the target detection information is not present in the other detection results, the method further includes:

[0031] The confidence level corresponding to the benchmark detection information is used as the reference confidence level;

[0032] When the reference confidence level is determined to be greater than the third sub-threshold, the benchmark detection information is determined as the candidate detection information of the object corresponding to the benchmark detection information in the image.

[0033] In one possible implementation, obtaining the annotation result corresponding to the image based on the candidate detection information corresponding to each object within the image includes:

[0034] For each object in the image, perform the following operations:

[0035] If an object has multiple candidate detection information, then the multiple candidate detection information are fused to obtain the annotation information of the object;

[0036] If an object has a candidate detection information, then the candidate detection information is used as the annotation information of the object;

[0037] The annotation information corresponding to each object in the image is determined as the annotation result corresponding to the image.

[0038] In one possible implementation, the candidate detection information includes the coordinates of a detection box that defines the corresponding object in the corresponding image;

[0039] The process of fusing the multiple candidate detection information to obtain the annotation information of the object includes:

[0040] The average coordinates of the detection boxes corresponding to the multiple candidate detection information are used as the coordinates of the final detection box. Based on the coordinates of the final detection box, the annotation information of the object is obtained; or

[0041] Based on the coordinates of the detection boxes corresponding to the multiple candidate detection information, the coordinates of the minimum bounding rectangle containing the multiple detection boxes are determined, and the annotation information of the object is obtained based on the coordinates of the minimum bounding rectangle.

[0042] In one possible implementation, before fusing the plurality of candidate detection information, the method further includes:

[0043] When the number of candidate detection information corresponding to the object is the same as the number of multiple detection models, and the confidence level of each candidate detection information is greater than the second confidence level threshold, the identifier of the object is set to a valid value.

[0044] After obtaining the annotation information of the object, and before determining the annotation information corresponding to each object in the image as the annotation result corresponding to the image, the method further includes:

[0045] When the identifier of an object is determined to be a valid value, the color of the marker box in the annotation information corresponding to the object is adjusted from the first color to the second color.

[0046] In one possible implementation, before determining the annotation information corresponding to each object in the image as the annotation result corresponding to the image, the method further includes:

[0047] For each annotation, perform the following operations:

[0048] For a given annotation, determine the Cross Union Ratio (CIOU) value between the given annotation and each other annotation;

[0049] For each CIOU value, perform the following operations:

[0050] For a CIOU value, if it is determined that the CIOU value is greater than the CIOU threshold, the annotation information corresponding to the lower confidence level among the two annotation information associated with the CIOU value is deleted.

[0051] Secondly, embodiments of this application provide a data annotation device based on multi-model fusion, comprising:

[0052] The acquisition module is used to acquire multiple images associated with the annotation task based on the annotation instructions for the annotation task;

[0053] The object detection module is used to perform object detection on each of the multiple images using multiple detection models associated with the annotation task, and to obtain multiple detection results corresponding to each image; wherein, each detection result corresponds to a detection model; the detection result includes at least one detection information corresponding to each object detected in the corresponding image; each detection information corresponds to a confidence level;

[0054] The multi-model fusion module is used to perform the following operations for each image: for an image, based on multiple detection results corresponding to the image and a first confidence threshold, to filter out candidate detection information corresponding to each object in the image; based on the candidate detection information corresponding to each object in the image, to obtain the annotation result corresponding to the image, wherein the annotation result includes the annotation information corresponding to each object in the image.

[0055] In one possible implementation, the multi-model fusion module is specifically used for:

[0056] For each of the multiple detection results, perform the following operations:

[0057] For a detection result, determine the baseline detection information in the detection result, wherein the baseline detection information is the unprocessed detection information in the detection result;

[0058] For each determined benchmark detection information, perform the following operations:

[0059] For a benchmark detection information, based on the intersection-union ratio (IOU) of the benchmark detection information and each target detection information in other detection results, the matching detection information corresponding to the benchmark detection information in each other detection result is determined, wherein the target detection information is the detection information that has the same detection category as the benchmark detection information but is not matched.

[0060] Based on the confidence of the benchmark detection information and the confidence of the candidate detection information that is greater than the IOU threshold among all matching detection information, the reference confidence of the object corresponding to the benchmark detection information is determined.

[0061] When the reference confidence level is determined to be greater than the first confidence threshold, each candidate detection information that is greater than the IOU threshold is determined as the candidate detection information of the object corresponding to the benchmark detection information in the image.

[0062] In one possible implementation, the multi-model fusion module is specifically used for:

[0063] Determine the IOU value of the benchmark detection information with each target detection information in other detection results;

[0064] The other detection information corresponding to the maximum IOU value in each of the other detection results is used as the matching detection information;

[0065] The multi-model fusion module is specifically used for:

[0066] Based on the IOU value between all matching detection information and the benchmark detection information, and the IOU value between every two matching detection information in all matching detection information, each candidate detection information with an IOU value greater than the threshold is selected from all matching detection information and the benchmark detection information.

[0067] The highest confidence level among the confidence levels corresponding to each candidate detection information is determined as the reference confidence level of the object corresponding to the benchmark detection information.

[0068] In one possible implementation, the first confidence threshold includes multiple sub-thresholds;

[0069] If the number of identified candidate detection information is equal to the total number of the plurality of detection models, then the first sub-threshold among the plurality of sub-thresholds is taken as the first confidence threshold.

[0070] If the number is less than the total number but greater than a preset value, then the second sub-threshold among the multiple sub-thresholds is used as the first confidence threshold.

[0071] If the number is equal to the preset value, then the third sub-threshold among the multiple sub-thresholds is taken as the first confidence threshold;

[0072] Wherein, the first sub-threshold is less than or equal to the second sub-threshold, and the second sub-threshold is less than the third sub-threshold.

[0073] In one possible implementation, if the target detection information is not present in the other detection results, the multi-model fusion module is further configured to:

[0074] The confidence level corresponding to the benchmark detection information is used as the reference confidence level;

[0075] When the reference confidence level is determined to be greater than the third sub-threshold, the benchmark detection information is determined as the candidate detection information of the object corresponding to the benchmark detection information in the image.

[0076] In one possible implementation, the multi-model fusion module is specifically used for:

[0077] For each object in the image, perform the following operations:

[0078] If an object has multiple candidate detection information, then the multiple candidate detection information are fused to obtain the annotation information of the object;

[0079] If an object has a candidate detection information, then the candidate detection information is used as the annotation information of the object;

[0080] The annotation information corresponding to each object in the image is determined as the annotation result corresponding to the image.

[0081] In one possible implementation, the candidate detection information includes the coordinates of a detection box that defines the corresponding object in the corresponding image;

[0082] The multi-model fusion module is specifically used for:

[0083] The average coordinates of the detection boxes corresponding to the multiple candidate detection information are used as the coordinates of the final detection box. Based on the coordinates of the final detection box, the annotation information of the object is obtained; or

[0084] Based on the coordinates of the detection boxes corresponding to the multiple candidate detection information, the coordinates of the minimum bounding rectangle containing the multiple detection boxes are determined, and the annotation information of the object is obtained based on the coordinates of the minimum bounding rectangle.

[0085] In one possible implementation, before fusing the multiple candidate detection information, the multi-model fusion module is further configured to:

[0086] When the number of candidate detection information corresponding to the object is the same as the number of multiple detection models, and the confidence level of each candidate detection information is greater than the second confidence level threshold, the identifier of the object is set to a valid value.

[0087] After obtaining the annotation information of the object, and before determining the annotation information corresponding to each object in the image as the annotation result corresponding to the image, the method further includes:

[0088] When the identifier of an object is determined to be a valid value, the color of the marker box in the annotation information corresponding to the object is adjusted from the first color to the second color.

[0089] In one possible implementation, before determining the annotation information corresponding to each object in the image as the annotation result corresponding to the image, the multi-model fusion module is further configured to:

[0090] For each annotation, perform the following operations:

[0091] For a given annotation, determine the Cross Union Ratio (CIOU) value between the given annotation and each other annotation;

[0092] For each CIOU value, perform the following operations:

[0093] For a CIOU value, if it is determined that the CIOU value is greater than the CIOU threshold, the annotation information corresponding to the lower confidence level among the two annotation information associated with the CIOU value is deleted.

[0094] Thirdly, embodiments of this application provide an electronic device, including:

[0095] Memory is used to store computer programs or instructions;

[0096] A processor for executing a computer program or instructions in the memory, such that the method described in any of the first aspects above is performed.

[0097] Fourthly, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor, enables the processor to perform the method described in any one of the first aspects above.

[0098] Furthermore, the technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of different implementation methods in the first aspect, and will not be repeated here. Attached Figure Description

[0099] Figure 1A This is a schematic diagram illustrating an optional application scenario of a data annotation method based on multi-model fusion in this application embodiment;

[0100] Figure 1B This is a schematic diagram of a human-computer interaction interface in a labeling platform according to an embodiment of this application;

[0101] Figure 1C This is a schematic diagram of the human-computer interaction interface in another annotation platform in this application embodiment;

[0102] Figure 2 This is a flowchart illustrating a data annotation method based on multi-model fusion in an embodiment of this application;

[0103] Figure 3 This is a flowchart illustrating a process for determining candidate detection information of an object corresponding to each detection information in the detection result, as described in an embodiment of this application.

[0104] Figure 4 This is a schematic diagram of a process for determining matching detection information in an embodiment of this application;

[0105] Figure 5 This is a schematic diagram of a process for determining the reference confidence level of an object corresponding to benchmark detection information in an embodiment of this application;

[0106] Figure 6 This is a schematic diagram of a process for determining the annotation information of each object in the image according to an embodiment of this application;

[0107] Figure 7 This is a schematic diagram illustrating the execution flow of a filtering box strategy in an embodiment of this application;

[0108] Figure 8 This is a schematic diagram of an image showing multiple bounding boxes for the same object in an embodiment of this application;

[0109] Figure 9 This is a schematic diagram comparing the data annotation results of single-model detection and multi-model fusion in the embodiments of this application;

[0110] Figure 10This is a schematic diagram of the logical architecture of a data annotation device based on multi-model fusion in an embodiment of this application;

[0111] Figure 11 This is a schematic diagram of the physical architecture of the electronic device in the embodiments of this application. Detailed Implementation

[0112] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0113] To help those skilled in the art better understand the technical solutions of this application, some terms used in the embodiments of this application are explained below.

[0114] (1) In the embodiments of this application, the term "and / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the related objects before and after are in an "or" relationship.

[0115] (2) The term “intersection over union (IOU)” in the embodiments of this application is used to calculate the ratio of the intersection and union of two detection boxes.

[0116] (3) In the embodiments of this application, the term "Complete Intersection over Union (CIOU)" fully considers the overlap area, center-point distance, and aspect ratio between pairwise detection boxes, taking into account the similarity between two detection boxes from three aspects: overlap area, center-point distance, and aspect ratio. In the embodiments of this application, CIOU can be expressed by the following formula:

[0117]

[0118]

[0119]

[0120] Where Distance_2 represents the Euclidean distance between the center points of the two detection boxes, and Distance_C represents the distance between the diagonals of the minimum bounding rectangles of the two detection boxes.

[0121] It should be noted that the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0122] The design concept of the embodiments of this application will be briefly introduced below.

[0123] This application relates to the field of image processing technology, and mainly to a data annotation method and electronic device based on multi-model fusion.

[0124] In the field of machine vision, massive amounts of collected images are used to train various models. Typically, before training a model, these massive images need to be labeled to obtain label information for training different models. Currently, conventional image labeling is mostly done manually, which suffers from low efficiency, slow speed, long processing time, and high cost.

[0125] In view of this, in order to solve the problems of low efficiency, slow speed, long time and high cost of conventional annotation methods, this application provides a data annotation method based on multi-model fusion. In this application embodiment, based on the annotation instruction for the annotation task, multiple images associated with the annotation task are obtained; using multiple detection models associated with the annotation task, target detection is performed on each of the multiple images to obtain multiple detection results corresponding to each image; wherein, each detection result corresponds to a detection model; the detection result includes at least one detection information corresponding to each object detected in the corresponding image; each detection information corresponds to a confidence level; for each image, the following operations are performed: for an image, based on the multiple detection results corresponding to the image and a first confidence level threshold, candidate detection information corresponding to each object in the image is selected; based on the candidate detection information corresponding to each object in the image, the annotation result corresponding to the image is obtained, wherein the annotation result includes the detection box annotation information corresponding to each object in the image.

[0126] In this embodiment, firstly, target detection is performed on multiple images associated with the annotation task based on each detection model to obtain detection results. Then, through a multi-model fusion decision-making method, the detection information in the multiple detection results corresponding to each image is filtered, and the filtered candidate detection information is fused to obtain the annotation result corresponding to each image. In this way, the precision and recall of the annotation result corresponding to each image are better than the prediction result of a single model. Annotators only need to perform secondary annotation based on this annotation result, and the annotation speed is better than the annotation speed based on single model prediction. At the same time, since multiple models are used to obtain the preliminary annotation result corresponding to each image, the annotation efficiency and speed are improved, the annotation time is shortened, and the annotation cost is reduced.

[0127] The data annotation method based on multi-model fusion in this application embodiment is applicable to electronic devices and is mainly used for offline tasks with an annotation platform deployed.

[0128] In some embodiments, the electronic device may be a smart terminal, a smart mobile terminal, a tablet computer, a laptop computer, a smart handheld device, a personal computer (PC), a computer, a smart screen, a personal digital assistant (PDA), etc.

[0129] In other embodiments, the electronic device may also be a server connected to a device with display functionality (e.g., a display device). For example, it may be a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed system that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server may be connected to the device with display functionality via wired or wireless means, and this application does not limit the connection method.

[0130] It is understood that this application does not limit the specific type of the aforementioned electronic device.

[0131] The following diagram, with reference to the accompanying drawings, illustrates an application scenario of an optional data annotation method based on multi-model fusion, using an electronic device as a server as an example. Figure 1A As shown, this application scenario includes a labeling terminal 10 with a labeling platform deployed, and a server 20 associated with the labeling platform.

[0132] The annotation platform provides users with a human-computer interaction interface, such as... Figure 1BAs shown. The annotation management personnel's terminal can construct annotation tasks through this human-computer interaction interface, such as selecting multiple associated images and multiple associated detection models for the annotation task to be initiated; after constructing the annotation task, the annotation management personnel can click on a preset area in the human-computer interaction interface (such as...). Figure 1B (The area where the "OK" button is shown) issues annotation instructions for this annotation task.

[0133] After receiving the annotation instruction sent by the annotation platform, the annotation management terminal sends the annotation instruction to the server 20. The server 20 calls the deployed multi-model fusion decision algorithm and executes a data annotation method based on multi-model fusion in this embodiment of the application to obtain the annotation result corresponding to each image associated with the annotation task. The annotation result includes the annotation information corresponding to each object in the corresponding image.

[0134] In some feasible embodiments, the annotation result can be a pre-annotation result. After the server 20 obtains the annotation result corresponding to each image associated with the annotation task, the annotator logged into this annotation platform can obtain the annotation result of each image associated with the annotation task through the human-computer interaction interface of the associated annotation terminal, based on the identification information of the annotation task, and display them respectively. Figure 1C As shown, the annotation result of an image is displayed in the human-computer interaction interface of any annotator's annotation terminal 10. The annotator can perform secondary annotation on the displayed annotation result of the image through the human-computer interaction interface to obtain the final annotation result of the image, and then obtain the final annotation result of each image associated with the annotation task.

[0135] It should be noted that in the embodiments of this application, each detection model can be trained based on a small amount of manually labeled data, or it can be a large pre-trained model. This application does not impose any specific limitations.

[0136] Of course, the methods provided in the embodiments of this application are not limited to... Figure 1A The application scenarios shown can also be used in other possible application scenarios, and the embodiments in this application are not limited thereto. It should also be noted that... Figure 1B , Figure 1C The human-computer interaction interface shown is only an example. In actual applications, the content displayed in the human-computer interaction interface can be customized according to actual needs. This application does not impose any specific limitations.

[0137] After introducing one optional application scenario of the embodiments of this application, the preferred embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

[0138] See Figure 2 As shown, see reference Figure 1A The application scenario shown in this application illustrates a data annotation method based on multi-model fusion. The specific process is as follows:

[0139] Step 200: Based on the annotation instructions for the annotation task, obtain multiple images associated with the annotation task.

[0140] In this embodiment, after the annotation platform initiates an annotation task, the annotation terminal receives the annotation instruction for the task and sends it to the server associated with the annotation platform. The server executes step 200, acquiring multiple images associated with the annotation task based on the annotation instruction. Simultaneously, it acquires multiple detection models associated with the annotation, as well as the detection categories of objects to be detected from the images, such as cars, pedestrians, zebra crossings, and buses.

[0141] Step 210: Using the multiple detection models associated with the annotation task, target detection is performed on each of the multiple images to obtain multiple detection results corresponding to each image; wherein, each detection result corresponds to a detection model; the detection result includes at least one detection information corresponding to each object detected in the corresponding image; each detection information corresponds to a confidence level.

[0142] In specific implementation, when executing step 210, the following operations are performed for each of the multiple images to obtain multiple detection results corresponding to each image: For an image (i.e., any one of the multiple images), multiple detection models associated with the annotation task are used to perform target detection on the objects in the image respectively, to obtain the detection information of each object in the image detected by each detection model, and the confidence level corresponding to the detection information. The detection information includes the coordinates of the detection box that defines the corresponding object in the image, and the confidence level is the probability that the object defined by the corresponding detection box belongs to the corresponding detection category.

[0143] Then, the following four strategies set in the embodiments of this application (i.e., the content executed in subsequent step 220) are adopted: 1. Retain box strategy; 2. Edit box strategy; 3. Confirmation box strategy; 4. Filter box strategy to determine the annotation result corresponding to each image.

[0144] Specifically, the detection results corresponding to multiple detection models are selected using a retain box strategy to determine which detection information needs to be retained and which needs to be deleted. Then, the selected detection information is edited using an edit box strategy. This is mainly used to fuse multiple detection information associated with an object, so that the label box in each annotation information more accurately and precisely defines the corresponding object.

[0145] Next, using the confirmation box strategy, the bounding boxes that each detection model has detected and that have a high confidence level are selected from the determined annotation information, and these annotation information is highlighted. That is, in the annotation interface of the annotation platform (i.e. the aforementioned human-computer interaction interface), they are displayed with a specific color. When annotators perform secondary annotation, they can skip these bounding boxes without checking or modifying them, assuming they are correct, thereby greatly improving the annotation speed and efficiency.

[0146] Finally, a filtering box strategy is used to remove redundant annotation information and retain only the necessary annotation information. This strategy is mainly for situations where an object is detected by different detection models into different detection categories, resulting in multiple annotation information for the same object. In this case, only one annotation information is retained. When annotators perform secondary annotation, they can only modify or delete the retained annotation information, reducing redundant operations on multiple annotation information for the same object.

[0147] In this embodiment, the above strategies ultimately output a fused annotation result, which exhibits superior precision and recall compared to the prediction results of a single model. Annotators only need to perform secondary annotation based on this result, improving annotation speed compared to annotation based on a single model prediction and shortening annotation time. Furthermore, employing multiple detection models to perform target detection on each image also improves the accuracy of each annotation in the resulting annotation, thereby enhancing the detection accuracy of the final model trained based on the annotation results.

[0148] After providing an overall overview of the processing logic in the embodiments of this application, step 220 in a data annotation method based on multi-model fusion in the embodiments of this application will be described in detail below.

[0149] Step 220: For each image, perform the following steps separately:

[0150] Step 2201: For an image, based on the multiple detection results corresponding to the image and the first confidence threshold, filter out the candidate detection information corresponding to each object in the image.

[0151] For specific implementation details, please refer to... Figure 3As shown, when performing step 2201, for each of the multiple detection results, the following steps are performed respectively:

[0152] Step 300: For a test result, determine the baseline test information in the test result, wherein the baseline test information is the unprocessed test information in the test result.

[0153] In this embodiment of the application, for a detection result corresponding to a detection model, when executing step 300, each unprocessed detection information in the detection result is used as a benchmark detection information.

[0154] Step 310: For each determined benchmark detection information, perform the following steps respectively:

[0155] Step 3101: For a benchmark detection information, based on the IOU value of the benchmark detection information and each target detection information in other detection results, determine the matching detection information corresponding to the benchmark detection information in each other detection result, wherein the target detection information is the detection information that has the same detection category as the benchmark detection information but is not matched.

[0156] In specific implementation, when performing step 3101, please refer to... Figure 4 As shown, the specific steps are as follows:

[0157] Step 400: Determine the IOU value of the benchmark detection information with each target detection information in other detection results.

[0158] In this embodiment of the application, when performing step 400, based on the detection boxes corresponding to each target detection information in the benchmark detection information and other detection results, the ratio of the intersection and union of the corresponding two detection boxes is calculated, and the IOU value corresponding to each target detection information in the benchmark detection information and other detection results is obtained.

[0159] Step 410: Take the other detection information corresponding to the maximum IOU value in each other detection result as the matching detection information.

[0160] In this embodiment of the application, when performing step 410, the following operations are performed for each other detection result: For an other detection result, the maximum IOU value is selected from the IOU values ​​corresponding to each detection result in the benchmark detection information and the other detection information corresponding to the maximum IOU value, and the other detection information corresponding to the maximum IOU value is determined as the matching detection information that matches the benchmark detection information.

[0161] Step 3102: Based on the confidence of the benchmark detection information and the candidate detection information that is greater than the IOU threshold among all matching detection information, determine the reference confidence of the object corresponding to the benchmark detection information.

[0162] In specific implementation, when performing step 3102, please refer to... Figure 5 As shown, the specific steps are as follows:

[0163] Step 500: Based on the IOU values ​​of all matching detection information and the benchmark detection information, and the IOU values ​​between every two matching detection information in all matching detection information, select each candidate detection information that is greater than the IOU threshold from all matching detection information and the benchmark detection information.

[0164] In specific implementation, when performing step 500, based on all matching detection information, the IOU value of the detection box corresponding to every two matching detection information is determined, and from the IOU value of the baseline detection information and all matching detection information, as well as the IOU value of the detection box corresponding to every two matching detection information, each candidate IOU value greater than the IOU threshold is selected, and the detection information corresponding to each candidate IOU value is used as candidate detection information.

[0165] Step 510: Determine the highest confidence level among the confidence levels corresponding to each candidate detection information as the reference confidence level of the object corresponding to the benchmark detection information.

[0166] In some feasible embodiments, when performing step 3101, if there is no target detection information in other detection results, the confidence level corresponding to the benchmark detection information is used as the reference confidence level of the object corresponding to the benchmark detection information. Then, it is determined whether the reference confidence level is greater than the first confidence level threshold. After determining that the reference confidence level is greater than the first confidence level threshold, the subsequent step 3103 is executed. If it is determined that the reference confidence level is less than or equal to the first confidence level threshold, the benchmark detection information is deleted.

[0167] Step 3103: When the reference confidence level is determined to be greater than the first confidence level threshold, each of the above-mentioned candidate detection information with an IOU value is determined as the candidate detection information of the object corresponding to the benchmark detection information in the image.

[0168] In this embodiment of the application, after obtaining the reference confidence level in step 3102, if it is determined that the reference confidence level is less than or equal to the first confidence level threshold, it indicates that the detection information detected by the model this time is likely a false detection. Then, the baseline detection information and each matching detection information in other detection results that match the baseline detection information are deleted.

[0169] In order to better filter the detection information detected by each detection model and retain more accurate detection information corresponding to each object in each image, in a preferred embodiment of this application, the first confidence threshold includes multiple sub-thresholds, and specific values ​​can be set for the multiple sub-thresholds included in the first confidence threshold according to the number of detection models used.

[0170] In a specific embodiment, the first confidence threshold can be determined based on the number of the determined candidate detection information (i.e., the detection information greater than the IOU threshold among the baseline detection information and all matching detection information).

[0171] In this embodiment of the application, depending on the number of determined candidate detection information, the following three cases are included, but not limited to:

[0172] Scenario 1: If the number of identified candidate detection information is equal to the total number of multiple detection models, that is, the IOU value between the baseline detection information and each matching detection information, as well as the IOU value between any two matching detection information, is greater than the IOU threshold, it indicates that the objects bounded by the baseline detection information and all matching detection information are the same real object, and this object is detected by all three models. In this case, the first sub-threshold (denoted as p1) among the multiple sub-thresholds is taken as the first confidence threshold. At this time, if the reference confidence p is greater than p1, it is retained, that is, the baseline detection information and all matching detection information are retained; if the reference confidence p is less than p1, it is not retained, that is, the baseline detection information and all matching detection information are deleted.

[0173] Scenario 2: If the number of identified candidate detection information is less than the total number of multiple detection models but greater than a preset value (e.g., 1), meaning that at least one of the IOU values ​​between the baseline detection information and each matched detection information, and between any two matched detection information, is greater than the IOU threshold, it indicates that the objects bounded by the baseline detection information and the matched detection information greater than the IOU threshold are the same real object, and this object has been detected by at least two models. In this case, the second sub-threshold (denoted as p2) among the multiple sub-thresholds is used as the first confidence threshold. At this time, if the reference confidence p is greater than p2, the detection information associated with the IOU threshold is retained. If there is any detection information other than the one associated with the IOU threshold among the baseline detection information and all matched detection information, the detection information is deleted. If the confidence p is less than the confidence threshold p2, it is not retained, that is, the baseline detection information and all matched detection information are deleted. Here, p2 is greater than or equal to p1.

[0174] Scenario 3: If the number of identified candidate detection information is equal to a preset value (e.g., 1), meaning the IOU value between the baseline detection information and each matching detection information, as well as the IOU values ​​of every two matching detection information, are all less than the IOU threshold, or if it is determined that no target detection information exists in other detection results, it indicates that there is no matching detection information that matches the baseline detection information. In this case, the third sub-threshold (denoted as p3) among multiple sub-thresholds is used as the first confidence threshold. If the reference confidence p is greater than p3, the baseline detection information is retained; if the reference confidence p is less than p3, it is not retained, i.e., the baseline detection information is deleted. Here, p3 is greater than p2. In practical applications, p3 can be set to a large value, representing that if only one model detects it, it can only be retained if its confidence is greater than this large p3.

[0175] The specific processing flow of step 2201 will be described in detail below with a specific example.

[0176] For example, consider the annotation task associating three detection models.

[0177] In this embodiment of the application, after executing step 210, the detection results corresponding to each image associated with the annotation task are obtained, that is, the three detection results are obtained by using these three detection models to perform target detection on objects of the preset detection categories in each image.

[0178] Let's assume the three detection models are referred to as Model 1, Model 2, and Model 3, respectively.

[0179] In specific implementation, when executing step 2201, the detection result corresponding to the first model is used as the benchmark detection result, and each detection information in the benchmark detection result is used as benchmark detection information. For each benchmark detection information, the following operations are performed:

[0180] Operation 1 (corresponding to step 400 above): For a benchmark detection information, determine the IOU value of the benchmark detection information with each other detection information in the second model and the third model that has the same detection category as the benchmark detection information but does not match. That is, calculate the intersection-union ratio of the detection boxes included in the benchmark detection information with the detection boxes included in each other detection information in the second model and the third model to obtain the corresponding IOU value.

[0181] Operation 2 (corresponding to step 410 above): Based on the IOU values ​​obtained in Operation 1, the other detection information corresponding to the maximum IOU value in the second model and the third model is determined as the matching detection information that matches the benchmark detection information in the second model and the third model, respectively, and is denoted as matching detection information 1 and matching detection information 2; wherein, the IOU value between the benchmark detection information and each matching detection information is denoted as max_iou12 and max_iou13, respectively.

[0182] At the same time, the IOU value between matching detection information 1 and matching detection information 2 is determined and denoted as max_iou23.

[0183] Then, based on max_iou12, max_iou13, and max_iou23, each candidate detection information with an IOU threshold greater than the baseline detection information, matching detection information 1, and matching detection information 2 is selected (i.e., corresponding to the aforementioned step 500); the maximum confidence among the confidence levels corresponding to each candidate detection information is determined as the reference confidence level of the object corresponding to the baseline detection information (i.e., corresponding to the aforementioned step 510).

[0184] Secondly, based on the number of candidate detection information, the specific value of the first confidence threshold is determined. Alternatively, the specific value of the first confidence threshold can be determined based on the number of values ​​greater than the IOU threshold among max_iou12, max_iou13, and max_iou23.

[0185] Specifically, if max_iou12, max_iou13, and max_iou23 are all greater than the IOU threshold (corresponding to the aforementioned case 1), it indicates that the objects bounded by the detection boxes included in these three detection information are the same object, that is, the object is detected by all three models. In this case, the first sub-threshold among the multiple sub-thresholds (denoted as p1) is taken as the first confidence threshold. At this time, the maximum confidence among the three prediction information is taken as the reference confidence of the object corresponding to the baseline detection information, denoted as p. If the reference confidence p is greater than p1, the three detection information are retained, that is, the baseline detection information, matching detection information 1, and matching detection information 2 are all retained and determined as candidate detection information. If the reference confidence p is less than p1, the three detection information are deleted, that is, the baseline detection information, matching detection information 1, and matching detection information 2 are deleted. If they are retained, the three detection information are subjected to an "edit box" operation, and the final annotation information is output.

[0186] If one or two of max_iou12, max_iou13, and max_iou23 are greater than the IOU threshold (corresponding to the aforementioned case two), it indicates that the object bounded by the detection boxes included in the baseline detection information and the matching detection information greater than the IOU threshold is the same object, that is, the object is detected by at least two models. In this case, the second sub-threshold (denoted as p2) among the multiple sub-thresholds is used as the first confidence threshold. At this time, the maximum confidence among the baseline detection information and the matching detection information greater than the IOU threshold is used as the reference confidence of the object corresponding to the baseline detection information, denoted as p2. If the reference confidence p is greater than p2, then the detection information associated with the IOU threshold is retained and identified as candidate detection information. If there is still detection information other than the detection information associated with the IOU threshold among the baseline detection information, matching detection information 1, and matching detection information 2, then the detection information is deleted. If it is less than the confidence threshold p2, then it is not retained, that is, the baseline detection information, matching detection information 1, and matching detection information 2 are deleted, where p2 is greater than or equal to p1. If it is retained, then the two detection information are edited and the final annotation information is output.

[0187] If max_iou12, max_iou13, and max_iou23 are all less than the IOU threshold, or if it is determined that there is no unmatched detection information (i.e., the aforementioned target detection information) in the detection results corresponding to the second and third models (corresponding to the aforementioned case three), it means that there is no matching detection information that matches the benchmark detection information. In this case, the third sub-threshold (denoted as p3) among the multiple sub-thresholds is used as the first confidence threshold. At this time, the confidence corresponding to the benchmark detection information is used as the reference confidence of the object corresponding to the benchmark detection information, denoted as p. If the reference confidence p is greater than p3, the benchmark detection information is retained and determined as candidate detection information. If the reference confidence p is less than p3, it is not retained, i.e., the benchmark detection information is deleted, where p3 is greater than p2. In practical applications, p3 can be set to a large value, representing that if only one model detects it, it can only be retained if its confidence is greater than this large p3.

[0188] After processing the detection results corresponding to the first model, the unprocessed detection information is used as the baseline detection information, based on the detection results corresponding to the second model. For each baseline detection information, the following operations are performed:

[0189] For a baseline detection information, calculate the IOU value one by one with the unprocessed detection information in the detection results of the third model. Find the detection information with the largest IOU value as the matching detection information 3 of the baseline detection information in the third model. The IOU value between the baseline detection information and the matching detection information 3 is still recorded as max_iou23.

[0190] Based on the relationship between max_iou23 and the IOU threshold, determine the specific value of the first confidence threshold.

[0191] If max_iou23 is greater than the IOU threshold (corresponding to the aforementioned case 2), it means that the objects bounded by the detection boxes included in the baseline detection information and the matching detection information 3 are the same real object, that is, the object has been detected by both models. In this case, the second sub-threshold (denoted as p2) among the multiple sub-thresholds is used as the first confidence threshold. At this time, the maximum confidence among the two detection information is used as the reference confidence of the object corresponding to the baseline detection information, denoted as p. If the reference confidence p is greater than p2, the baseline detection information and the matching detection information 3 are retained and determined as candidate detection information. If the reference confidence p is less than p2, they are not retained, that is, the baseline detection information and the matching detection information 3 are deleted. If they are retained, the two detection information are "edited" and the final annotation information is output.

[0192] If max_iou23 is less than the IOU threshold, or if it is determined that there is no unmatched detection information (i.e., the aforementioned target detection information) in the detection results corresponding to the third model (corresponding to the aforementioned case three), it means that the benchmark detection information has no matching detection information. In this case, the third sub-threshold (denoted as p3) among the multiple sub-thresholds is used as the first confidence threshold. At this time, the confidence corresponding to the benchmark detection information is used as the reference confidence p of the object corresponding to the benchmark detection information. If the reference confidence p is greater than p3, the benchmark detection information is retained and determined as candidate detection information. If the reference confidence p is less than the threshold p3, it is not retained, that is, the benchmark detection information is deleted.

[0193] After processing the detection results corresponding to the second model, the unprocessed detection information is used as the baseline detection information, based on the detection results corresponding to the third model. For each baseline detection information, the following operations are performed:

[0194] For a benchmark detection information, the third sub-threshold (denoted as p3) among multiple sub-thresholds is used as the first confidence threshold (corresponding to the aforementioned case 3). At this time, the confidence corresponding to the benchmark detection information is used as the reference confidence of the object corresponding to the benchmark detection information, denoted as p. If the reference confidence p is greater than p3, the benchmark detection information is retained and determined as a candidate detection information. If the reference confidence p is less than p3, it is not retained, that is, the benchmark detection information is deleted.

[0195] In this way, through the above processing flow, the candidate detection information corresponding to each object in the image can be obtained, which prepares for the subsequent "edit box" operation.

[0196] Step 2202: Based on the candidate detection information corresponding to each object in the image, obtain the annotation result corresponding to the image, wherein the annotation result includes the annotation information corresponding to each object in the image.

[0197] In specific implementation, when performing step 2202, please refer to... Figure 6 As shown, the specific steps are as follows:

[0198] Step 600: For each object in the image, perform the following steps:

[0199] Step 6001: If an object has multiple candidate detection information, then the multiple candidate detection information are fused to obtain the annotation information of the object.

[0200] In specific implementation, after executing step 2201 and before executing step 6001, if it is determined that the number of candidate detection information corresponding to the object is the same as the number of multiple detection models, and the confidence level corresponding to each candidate detection information is greater than the second confidence threshold, then the identifier of the object is set to a valid value. This identifier is used to set a specific color for the fused bounding box of the object, so that annotators do not need to modify this type of annotation information during secondary annotation, thereby greatly improving annotation speed and efficiency.

[0201] In this embodiment of the application, when performing step 6001, any of the following fusion methods can be used to obtain the annotation information of the object:

[0202] Method 1: The average coordinates of the detection boxes corresponding to multiple candidate detection information are used as the coordinates of the final detection box. Based on the coordinates of the final detection box, the annotation information of the object is obtained.

[0203] In this embodiment, the candidate detection information includes the coordinates of the detection box that defines the corresponding object in the corresponding image. Each detection box is represented by its center point, width, and height. The center point is denoted as (x...). i ,y i ), with width and height respectively w i ,h i .

[0204] Therefore, when using method one, the average value of the center points of all detection boxes is taken as the center point of the final fused detection box, denoted as . The average width and height of all detection boxes are taken as the width of the final fused detection box. high In this embodiment of the application, the coordinates of the final detection box after fusion can be represented by the following formula, that is, the coordinates of the marker box included in the annotation information of object i:

[0205]

[0206]

[0207]

[0208]

[0209] In the above formula, i is the object index, and n is the total number of objects contained in the corresponding image.

[0210] Method 2: Based on the coordinates of the detection boxes corresponding to multiple candidate detection information, determine the coordinates of the minimum bounding rectangle containing multiple detection boxes, and obtain the annotation information of the object based on the coordinates of the minimum bounding rectangle.

[0211] When using method two, for any object, the coordinates of the top-left corner of each detection box are (x... 1j ,y 1j The coordinates of the lower right corner are (x... 2j ,y 2j ), where j represents the j-th predicted bounding box of the corresponding object; then, the coordinates of the top-left corner of the fused detection box (i.e., the aforementioned minimum bounding rectangle) are used as The coordinates of the lower right corner are indicated by... Therefore, in this embodiment of the application, the coordinates of the smallest bounding rectangle after fusion can be represented by the following formula:

[0212]

[0213]

[0214] Step 6002: If an object has a candidate detection information, then use the candidate detection information as the annotation information of the object.

[0215] In this embodiment of the application, after obtaining the annotation information of the object in step 600, before determining the annotation information corresponding to each object in the image as the annotation result corresponding to the image in subsequent step 610, it is necessary to adjust the color of the marker box in the annotation information corresponding to the object from the first color to the second color when determining that the identifier of the object is a valid value. The first color is the default color of the marker box, and the second color is the aforementioned specific color.

[0216] For example, let's take the annotation task as an example of associating three detection models.

[0217] In this embodiment of the application, when it is determined that a certain benchmark detection information meets condition one, that is, the detection box contained in the benchmark detection information is detected by each model, it is determined whether the confidence of the three detection information of the object corresponding to the benchmark detection information in each detection model is greater than the second confidence threshold. The second confidence threshold is greater than the first confidence threshold. Preferably, the second confidence threshold is greater than the third sub-threshold. If all of them are greater than the second confidence threshold, the identifier of the object corresponding to the benchmark detection information is set to a valid value, such as sure_flag = 1.

[0218] In this embodiment of the application, before executing step 610, when sorting out each fused annotation information, if the identifier of a certain object is determined to be a valid value, that is, if the sure_flag of the object is determined to be 1, then the color of the marker box in the annotation information corresponding to the object is adjusted from the first color to the second color. Here, the first color is the default color corresponding to the corresponding detection category, and the second color is the aforementioned specific color. In this way, the objects corresponding to the benchmark detection information that meet the conditions of case one can be distinguished by the color of the marker box. Through this operation, the annotation speed of the annotators can be further improved, and the annotation work efficiency can be improved.

[0219] In this embodiment of the application, before performing step 610, refer to... Figure 7 As shown, the following steps also need to be performed:

[0220] Step 700: For each annotation, perform the following operations: For a single annotation, determine the CIOU value between that annotation and each other annotation.

[0221] After implementing the above strategy, the following problems may arise: For some objects, different detection models may detect different categories. Therefore, after implementing the above strategy, the same object may have two almost identical bounding boxes, but with different detection categories, such as... Figure 8 As shown, for the bus on the right side of the image, after applying the aforementioned retain box strategy, edit box strategy, and confirm box strategy, the bus has two annotation pieces of information, namely... Figure 8 The “3” and “9” shown in the image are a label (i.e. Figure 8 The "3" shown includes a yellow-colored marker box with the detection category "car"; another annotation information (i.e. Figure 8 The "9" shown is marked with an orange box, and the detection category is "bus".

[0222] If these two annotations are not further processed, the annotators will need to perform another deletion operation when annotating again, which reduces the annotation speed and increases the annotation time. Therefore, in this embodiment, only one annotation is needed for the same object. This allows the annotators to check and correct based on one annotation. If the detection category is incorrect, only the detection category needs to be modified, thereby reducing the annotation workload and improving the annotation speed.

[0223] Based on this, after obtaining the annotation information corresponding to each object in step 600, the aforementioned filter box strategy is executed before step 610.

[0224] In practice, for the annotation information retained after applying the edit box strategy to multiple detection models, a filtering operation is performed, i.e., step 700 is executed:

[0225] For each annotation, the CIOU value of the bounding box included in that annotation and each of the other annotations is calculated using the following formula:

[0226]

[0227]

[0228]

[0229] Where Distance_2 represents the Euclidean distance between the center points of the two marker boxes, and Distance_C represents the distance between the diagonals of the smallest bounding rectangle of the two marker boxes.

[0230] Step 710: For each CIOU value, perform the following operations: For a CIOU value, when it is determined that the CIOU value is greater than the CIOU threshold, delete the annotation information corresponding to the smaller confidence level among the two annotation information associated with the CIOU value.

[0231] In this embodiment of the application, when performing step 710, the following operations are performed for each CIOU value: For a CIOU value, the CIOU value is compared with the CIOU threshold. If the CIOU value is greater than the CIOU threshold, then for these two labeling information, the labeling information with higher confidence is retained, that is, the labeling information with lower confidence is deleted.

[0232] In this way, by using CIOU values, the overlap area, center-to-center distance, and aspect ratio between pairwise bounding boxes can be fully considered. Ultimately, objects selected by two bounding boxes with CIOU values ​​greater than a certain threshold (CIOU threshold) are considered the same object, filtering out labels with lower confidence and retaining only the label with higher confidence. Considering the similarity between bounding boxes from three aspects—overlap area, center-to-center distance, and aspect ratio—and filtering out redundant label information can effectively improve the accuracy of the model's pre-labeled bounding boxes.

[0233] See also Figure 8 As shown, by implementing the filtering box strategy, if a label is found, only one label is retained, i.e., only one label box is kept; if the label corresponding to "car" is retained (i.e., ... Figure 8 If the labeling is "3" as shown, then when the labeler performs secondary labeling, they only need to correct the detection category. If the labeling information corresponding to "bus" is retained (i.e., Figure 8 If the symbol “9” is shown, the annotator will not need to perform any further operations when making secondary annotations, which will further save annotation time and improve annotation speed.

[0234] Step 610: Determine the annotation information corresponding to each object in the image as the annotation result for the image.

[0235] Figure 9 This is a schematic diagram comparing single-model detection and data annotation results based on multi-model fusion in an embodiment of this application. In this embodiment, see [reference needed]. Figure 9 As shown, taking the annotation task as an example, which involves associating three detection models, the corresponding detection results are obtained by performing detection on the same image; among them, Figure 9 The top left, top right, and bottom left corners of the image represent the detection results of the three individual models, while the bottom right corner shows the annotation results based on multi-model fusion.

[0236] By comparing the results of single-model and multi-model fusion-based annotations, it is clear that the multi-model fusion-based annotation results combine the strengths and compensate for the weaknesses of each individual model's detection results, such as... Figure 9As shown, the single-model detection results in the upper left corner detect only three objects, all classified as "car". The single-model detection results in the upper right corner detect four objects, classified as "rider", "rider", "bus", and "person". Only one of these objects appears in the single-model detection results in the upper left corner, but the two have different classifications: one is "car" and the other is "bus". The single-model detection results in the lower left corner detect eight objects, classified as "car", "car", "rider", "other", "rider", "car", "rider", and "person". Three of these objects appear in the single-model detection results in the upper left corner and have the same classification, while four objects appear in the upper right corner. In the single-model detection results shown in the upper left corner, three detection categories are consistent, while the other detection category is inconsistent. However, the annotation information based on multi-model fusion contains seven annotations, with detection categories in the order of "car", "car", "rider", "other", "rider", "car", and "person". Specifically, the annotations highlight the four objects in the single-model detection results shown in the upper left corner that were missed by the detection categories "rider", "other", "rider", and "person", and the three objects in the single-model detection results shown in the upper right corner that were missed by the detection categories "car", "car", and "other". The falsely detected object with the detection category "bus" was deleted, as was the falsely detected object with the detection category "rider" in the single-model detection results shown in the lower left corner. Therefore, the data annotation method based on multi-model fusion in this embodiment can further improve the accuracy of the annotation information.

[0237] The data annotation method based on multi-model fusion in this application can achieve better results than the single-model method, which meets the reliability characteristic of trustworthiness. This method is applicable to all detection models (algorithms). When a user receives a new annotation task, they only need to pre-train multiple detection models to use the aforementioned data annotation method based on multi-model fusion, which meets the generalization characteristic of trustworthiness. Furthermore, the data annotation method based on multi-model fusion in this application can achieve the same results in different operating environments, which meets the reproducibility characteristic of trustworthiness. It maintains superior performance compared to the single-model method under any circumstances, which meets the robustness characteristic of trustworthiness. It can be intervened by other intelligent agents during the inference process, which meets the controllability characteristic of trustworthiness. Furthermore, the method proposed in this application also meets the controllability characteristic of trustworthiness.

[0238] Based on the same inventive concept, see [reference] Figure 10As shown in the figure, this application provides a data annotation device based on multi-model fusion, including:

[0239] The acquisition module 1010 is used to acquire multiple images associated with the annotation task based on the annotation instructions for the annotation task;

[0240] The target detection module 1020 is used to perform target detection on each of the multiple images using multiple detection models associated with the annotation task, and obtain multiple detection results corresponding to each image; wherein, each detection result corresponds to a detection model; the detection result includes at least one detection information corresponding to each object detected in the corresponding image; each detection information corresponds to a confidence level;

[0241] The multi-model fusion module 1030 is used to perform the following operations for each image: for an image, based on multiple detection results corresponding to the image and a first confidence threshold, to filter out candidate detection information corresponding to each object in the image; based on the candidate detection information corresponding to each object in the image, to obtain the annotation result corresponding to the image, wherein the annotation result includes the annotation information corresponding to each object in the image.

[0242] In one possible implementation, the multi-model fusion module 1030 is specifically used for:

[0243] For each of the multiple detection results, perform the following operations:

[0244] For a detection result, determine the baseline detection information in the detection result, wherein the baseline detection information is the unprocessed detection information in the detection result;

[0245] For each determined benchmark detection information, perform the following operations:

[0246] For a benchmark detection information, based on the intersection-union ratio (IOU) of the benchmark detection information and each target detection information in other detection results, the matching detection information corresponding to the benchmark detection information in each other detection result is determined, wherein the target detection information is the detection information that has the same detection category as the benchmark detection information but is not matched.

[0247] Based on the confidence of the benchmark detection information and the confidence of the candidate detection information that is greater than the IOU threshold among all matching detection information, the reference confidence of the object corresponding to the benchmark detection information is determined.

[0248] When the reference confidence level is determined to be greater than the first confidence threshold, each candidate detection information that is greater than the IOU threshold is determined as the candidate detection information of the object corresponding to the benchmark detection information in the image.

[0249] In one possible implementation, the multi-model fusion module 1030 is specifically used for:

[0250] Determine the IOU value of the benchmark detection information with each target detection information in other detection results;

[0251] The other detection information corresponding to the maximum IOU value in each of the other detection results is used as the matching detection information;

[0252] The multi-model fusion module 1030 is specifically used for:

[0253] Based on the IOU value between all matching detection information and the benchmark detection information, and the IOU value between every two matching detection information in all matching detection information, each candidate detection information with an IOU value greater than the threshold is selected from all matching detection information and the benchmark detection information.

[0254] The highest confidence level among the confidence levels corresponding to each candidate detection information is determined as the reference confidence level of the object corresponding to the benchmark detection information.

[0255] In one possible implementation, the first confidence threshold includes multiple sub-thresholds;

[0256] If the number of identified candidate detection information is equal to the total number of the plurality of detection models, then the first sub-threshold among the plurality of sub-thresholds is taken as the first confidence threshold.

[0257] If the number is less than the total number but greater than a preset value, then the second sub-threshold among the multiple sub-thresholds is used as the first confidence threshold.

[0258] If the number is equal to the preset value, then the third sub-threshold among the multiple sub-thresholds is taken as the first confidence threshold;

[0259] Wherein, the first sub-threshold is less than or equal to the second sub-threshold, and the second sub-threshold is less than the third sub-threshold.

[0260] In one possible implementation, if the target detection information is not present in the other detection results, the multi-model fusion module 1030 is further configured to:

[0261] The confidence level corresponding to the benchmark detection information is used as the reference confidence level;

[0262] When the reference confidence level is determined to be greater than the third sub-threshold, the benchmark detection information is determined as the candidate detection information of the object corresponding to the benchmark detection information in the image.

[0263] In one possible implementation, the multi-model fusion module 1030 is specifically used for:

[0264] For each object in the image, perform the following operations:

[0265] If an object has multiple candidate detection information, then the multiple candidate detection information are fused to obtain the annotation information of the object;

[0266] If an object has a candidate detection information, then the candidate detection information is used as the annotation information of the object;

[0267] The annotation information corresponding to each object in the image is determined as the annotation result corresponding to the image.

[0268] In one possible implementation, the candidate detection information includes the coordinates of a detection box that defines the corresponding object in the corresponding image;

[0269] The multi-model fusion module 1030 is specifically used for:

[0270] The average coordinates of the detection boxes corresponding to the multiple candidate detection information are used as the coordinates of the final detection box. Based on the coordinates of the final detection box, the annotation information of the object is obtained; or

[0271] Based on the coordinates of the detection boxes corresponding to the multiple candidate detection information, the coordinates of the minimum bounding rectangle containing the multiple detection boxes are determined, and the annotation information of the object is obtained based on the coordinates of the minimum bounding rectangle.

[0272] In one possible implementation, before fusing the multiple candidate detection information, the multi-model fusion module 1030 is further configured to:

[0273] When the number of candidate detection information corresponding to the object is the same as the number of multiple detection models, and the confidence level of each candidate detection information is greater than the second confidence level threshold, the identifier of the object is set to a valid value.

[0274] After obtaining the annotation information of the object, and before determining the annotation information corresponding to each object in the image as the annotation result corresponding to the image, the method further includes:

[0275] When the identifier of an object is determined to be a valid value, the color of the marker box in the annotation information corresponding to the object is adjusted from the first color to the second color.

[0276] In one possible implementation, before determining the annotation information corresponding to each object in the image as the annotation result corresponding to the image, the multi-model fusion module 1030 is further configured to:

[0277] For each annotation, perform the following operations:

[0278] For a given annotation, determine the Cross Union Ratio (CIOU) value between the given annotation and each other annotation;

[0279] For each CIOU value, perform the following operations:

[0280] For a CIOU value, if it is determined that the CIOU value is greater than the CIOU threshold, the annotation information corresponding to the lower confidence level among the two annotation information associated with the CIOU value is deleted.

[0281] Based on the same inventive concept, see [reference] Figure 11 As shown in the figure, an electronic device is provided in this application embodiment, including:

[0282] Memory 111 is used to store computer programs or instructions;

[0283] Processor 112 is configured to execute computer programs or instructions in the memory 111 such that any of the methods described in the above embodiments is performed.

[0284] The processor 112 may include one or more central processing units (CPUs) or digital processing units, etc., for executing computer programs or instructions in the memory 111, such that any of the methods described in the above embodiments is executed.

[0285] It should be noted that the specific connection medium between the memory 111 and the processor 112 is not limited in the embodiments of this application. The embodiments of this application... Figure 11 In this diagram, the memory 111 and processor 112 are connected via bus 113. The connections between other components are merely illustrative and not intended to be limiting. Bus 113 can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0286] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium that, when executed by a processor, enables the processor to perform any of the methods described in the above embodiments. Since the principle by which the above computer-readable storage medium solves the problem is similar to that of a data annotation method based on multi-model fusion, the implementation of the above computer-readable storage medium can be found in the implementation of the method; repeated details will not be elaborated further.

[0287] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0288] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0289] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more processes in a flowchart and / or one or more blocks in a block diagram.

[0290] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0291] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data annotation method based on multi-model fusion, characterized in that, include: Based on the annotation instructions for the annotation task, obtain multiple images associated with the annotation task; Using multiple detection models associated with the annotation task, target detection is performed on each of the multiple images to obtain multiple detection results corresponding to each image; wherein, each detection result corresponds to a detection model; the detection result includes at least one detection information corresponding to each object detected in the corresponding image; each detection information corresponds to a confidence level; For each of the images, perform the following operations: For an image, based on multiple detection results corresponding to the image and a first confidence threshold, candidate detection information corresponding to each object in the image is selected. Based on the candidate detection information corresponding to each object in the image, the annotation result corresponding to the image is obtained, wherein the annotation result includes the annotation information corresponding to each object in the image; The step of filtering candidate detection information for each object in an image based on multiple detection results corresponding to the image and a first confidence threshold includes: For each of the multiple detection results, perform the following operations: For a detection result, determine the baseline detection information in the detection result, wherein the baseline detection information is the unprocessed detection information in the detection result; For each determined benchmark detection information, perform the following operations: For a benchmark detection information, based on the intersection-union ratio (IOU) of the benchmark detection information and each target detection information in other detection results, the matching detection information corresponding to the benchmark detection information in each other detection result is determined, wherein the target detection information is the detection information that has the same detection category as the benchmark detection information but is not matched. Based on the confidence of the benchmark detection information and the confidence of the candidate detection information that is greater than the IOU threshold among all matching detection information, the reference confidence of the object corresponding to the benchmark detection information is determined. When the reference confidence level is determined to be greater than the first confidence threshold, each candidate detection information that is greater than the IOU threshold is determined as the candidate detection information of the object corresponding to the benchmark detection information in the image. The first confidence threshold includes multiple sub-thresholds; If the number of identified candidate detection information is equal to the total number of the plurality of detection models, then the first sub-threshold among the plurality of sub-thresholds is taken as the first confidence threshold. If the number is less than the total number but greater than a preset value, then the second sub-threshold among the multiple sub-thresholds is used as the first confidence threshold. If the number is equal to the preset value, then the third sub-threshold among the multiple sub-thresholds is taken as the first confidence threshold; Wherein, the first sub-threshold is less than or equal to the second sub-threshold, and the second sub-threshold is less than the third sub-threshold.

2. The method as described in claim 1, characterized in that, The step of determining the matching detection information corresponding to the benchmark detection information in each of the other detection results based on the intersection-union ratio (IOU) of the benchmark detection information and each target detection information in the other detection results includes: Determine the IOU value of the benchmark detection information with each target detection information in other detection results; The other detection information corresponding to the maximum IOU value in each of the other detection results is used as the matching detection information; The step of determining the reference confidence level of the object corresponding to the benchmark detection information based on the confidence level of the benchmark detection information and the candidate detection information greater than the IOU threshold among all matched detection information includes: Based on the IOU value between all matching detection information and the benchmark detection information, and the IOU value between every two matching detection information in all matching detection information, each candidate detection information with an IOU value greater than the threshold is selected from all matching detection information and the benchmark detection information. The highest confidence level among the confidence levels corresponding to each candidate detection information is determined as the reference confidence level of the object corresponding to the benchmark detection information.

3. The method as described in claim 1, characterized in that, If the target detection information is not found in the other detection results, the method further includes: The confidence level corresponding to the benchmark detection information is used as the reference confidence level; When the reference confidence level is determined to be greater than the third sub-threshold, the benchmark detection information is determined as the candidate detection information of the object corresponding to the benchmark detection information in the image.

4. The method according to any one of claims 1-3, characterized in that, The step of obtaining the annotation result corresponding to the image based on the candidate detection information corresponding to each object in the image includes: For each object in the image, perform the following operations: If an object has multiple candidate detection information, then the multiple candidate detection information are fused to obtain the annotation information of the object; If an object has a candidate detection information, then the candidate detection information is used as the annotation information of the object; The annotation information corresponding to each object in the image is determined as the annotation result corresponding to the image.

5. The method as described in claim 4, characterized in that, The candidate detection information includes the coordinates of the detection box that defines the corresponding object in the corresponding image; The process of fusing the multiple candidate detection information to obtain the annotation information of the object includes: The average coordinates of the detection boxes corresponding to the multiple candidate detection information are used as the coordinates of the final detection box. Based on the coordinates of the final detection box, the annotation information of the object is obtained; or Based on the coordinates of the detection boxes corresponding to the multiple candidate detection information, the coordinates of the minimum bounding rectangle containing the multiple detection boxes are determined, and the annotation information of the object is obtained based on the coordinates of the minimum bounding rectangle.

6. The method as described in claim 4, characterized in that, Before fusing the multiple candidate detection information, the method further includes: When the number of candidate detection information corresponding to the object is the same as the number of multiple detection models, and the confidence level of each candidate detection information is greater than the second confidence level threshold, the identifier of the object is set to a valid value. After obtaining the annotation information of the object, and before determining the annotation information corresponding to each object in the image as the annotation result corresponding to the image, the method further includes: When the identifier of an object is determined to be a valid value, the color of the marker box in the annotation information corresponding to the object is adjusted from the first color to the second color.

7. The method as described in claim 4, characterized in that, Before determining the annotation information corresponding to each object in the image as the annotation result corresponding to the image, the method further includes: For each annotation, perform the following operations: For a given annotation, determine the Cross Union Ratio (CIOU) value between the given annotation and each other annotation; For each CIOU value, perform the following operations: For a CIOU value, if it is determined that the CIOU value is greater than the CIOU threshold, the annotation information corresponding to the lower confidence level among the two annotation information associated with the CIOU value is deleted.

8. An electronic device, characterized in that, include: Memory is used to store computer programs or instructions; A processor for executing a computer program or instructions in the memory such that the method described in any one of claims 1-7 is performed.

Images