Consistency labeling decision-making method based on multiple algorithms
By employing a multi-algorithm consensus annotation decision method, consensus weights are calculated using intersection-union ratio (IoU) and similarity. The annotation results of vehicle image data are dynamically evaluated and weighted, solving the problem of insufficient robustness of single-algorithm models in image annotation and achieving higher accuracy and robustness in image annotation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-08
- Publication Date
- 2026-03-13
AI Technical Summary
Existing single-algorithm models struggle to maintain high accuracy across all scenarios in automated image annotation, especially in object detection tasks. This is due to limitations in model architecture, training data, or task scenarios, resulting in insufficient robustness and generalization ability.
A consensus annotation decision method based on multiple algorithms is adopted. Three annotation algorithm models (YOLOv8, Faster R-CNN, and DETR) are used to annotate vehicle image data. The consensus weight is calculated using the intersection-union ratio (IoU) and similarity. The consistency and confidence of the algorithms are dynamically evaluated and weighted to obtain the final annotation results.
It effectively identifies and handles discrepancies between models, reduces annotation errors, and improves the accuracy and robustness of annotation, making it suitable for various image annotation algorithms.
Smart Images
Figure CN121661618A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image annotation, and more particularly to a consensus annotation decision method based on multiple algorithms. Background Technology
[0002] Currently, automated image annotation techniques mainly rely on deep learning and computer vision models, such as YOLO and Faster R-CNN for object detection, and U-Net and Mask R-CNN for segmentation. While these models perform well on specific datasets, their performance is significantly constrained by multiple factors.
[0003] First, model biases are prevalent: CNN-based models excel at capturing local texture features but are weak in understanding global context; while Transformer-based models, although capable of modeling long-range dependencies, require large amounts of data and are prone to unstable predictions when insufficiently trained. Second, inherent task challenges are difficult to overcome: factors such as significant differences in object scale within images, partial occlusion, lighting variations, and inter-class similarity make it difficult for a single model to maintain robustness across all scenarios. Finally, there is strong data dependence: the performance of a model is highly dependent on the distribution of its training data; its generalization ability drops sharply when faced with out-of-distribution samples or rare classes.
[0004] In summary, in the field of automated image annotation, especially in object detection tasks, a single algorithm model often struggles to maintain high accuracy in all situations due to limitations in its own architecture, training data, or task scenario. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a consistency labeling decision method based on multiple algorithms.
[0006] The objective of this invention is achieved through the following technical solution: In its first aspect, this invention provides a consensus labeling decision method based on multiple algorithms, comprising the following steps: Acquire images captured by the vehicle's onboard camera; The image is input into three annotation algorithm models to obtain output results including annotation bounding boxes, annotation categories, and annotation confidence scores; Using the intersection-union ratio (IoU) of the labeled bounding boxes as the matching rule, the same object detected by different labeling algorithm models is matched; Calculate the similarity between the outputs of the pairwise labeling algorithm models, and obtain the consensus weight by using the average similarity with other labeling algorithm models; The final annotation result is obtained by performing weighted calculation on each object based on the consensus weight.
[0007] Furthermore, the labeling categories include pedestrians, vehicles, bicycles, and traffic signs.
[0008] Furthermore, the three annotation algorithm models include YOLOv8, Faster R-CNN, and DETR.
[0009] Furthermore, the bounding box is a rectangle with the format [x,y,width,height], where x and y represent the coordinates of the top-left corner of the rectangle, width represents the width of the rectangle, and height represents the height of the rectangle. The intersection-union ratio (IoU) of two bounding boxes is the area of their intersection divided by the area of their union. If the intersection-union ratio (IoU) of the bounding boxes of objects detected by different annotation algorithm models is greater than 0.5, they are considered to be the same object.
[0010] Furthermore, the similarity S between the outputs of the pairwise labeling algorithm model is calculated in the following ways: In the formula, Ai and Aj represent the i-th and j-th annotation algorithm models, respectively, and the average matching quality represents the average intersection-union ratio (IoU) of the matched objects in the i-th and j-th annotation algorithm models.
[0011] Furthermore, the step of obtaining the consensus weight by averaging the similarity with other annotation algorithm models includes: Calculate the consensus average similarity between each annotation algorithm model and other annotation algorithm models; The consensus weight is obtained by normalizing the average value of all consensus values.
[0012] Furthermore, the step of performing weighted summation on each object based on the consensus weight to obtain the final annotation result includes: The weighted bounding box is obtained by weighting the bounding box coordinates of each object in each annotation algorithm model with the consensus weight. The weighted bounding box is obtained by weighting the confidence score and consensus weight of each object in each annotation algorithm model. The weighted bounding box and the weighted bounding box are used as the final annotation results.
[0013] The beneficial effects of this invention are: In an exemplary embodiment of the present invention, three annotation algorithm models are used together to annotate vehicle image data. By dynamically evaluating the spatial consistency and confidence of the recognition results of different annotation algorithms on specific samples, they are intelligently assigned differentiated decision weights, thereby effectively identifying and handling serious discrepancies between models (such as missing detection of a certain object part of the model or large positioning differences), automatically filtering out uncertain annotations, and reducing annotation errors caused by algorithm blind spots or occasional errors. Attached Figure Description
[0014] Figure 1 A flowchart of a consensus labeling decision method based on multiple algorithms provided as an exemplary embodiment of the present invention. Detailed Implementation
[0015] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0016] See Figure 1 , Figure 1 The flowchart illustrates a consensus labeling decision method based on multiple algorithms provided by an exemplary embodiment of the present invention, including the following steps: Acquire images captured by the vehicle's onboard camera; The image is input into three annotation algorithm models to obtain output results including annotation bounding boxes, annotation categories, and annotation confidence scores; Using the intersection-union ratio (IoU) of the labeled bounding boxes as the matching rule, the same object detected by different labeling algorithm models is matched; Calculate the similarity between the outputs of the pairwise labeling algorithm models, and obtain the consensus weight by using the average similarity with other labeling algorithm models; The final annotation result is obtained by performing weighted calculation on each object based on the consensus weight.
[0017] Specifically, in this exemplary embodiment, three annotation algorithm models are used together to annotate vehicle image data. By dynamically evaluating the spatial consistency and confidence of the recognition results of different annotation algorithms on specific samples, they are intelligently assigned differentiated decision weights, thereby effectively identifying and handling serious discrepancies between models (such as missing detection of a certain object part of the model or large positioning differences), automatically filtering out uncertain annotations, and reducing annotation errors caused by algorithm blind spots or occasional errors.
[0018] Furthermore, this method has broad applicability and can be applied to almost all types of image annotation algorithms. Key success factors include: (1) Algorithm diversity: different algorithms with different principles and advantages can be selected; (2) Appropriate similarity measurement: an effective comparison method is designed according to the characteristics of the task; (3) Dynamic weight adjustment mechanism: to ensure that the algorithm that is consistent with the mainstream opinion gets more discourse power; (4) Anomaly handling: special handling procedures are set up for situations with great disagreement.
[0019] The following will describe the specific implementation of each step, wherein: the annotation categories include pedestrians, vehicles, bicycles, and traffic signs; and the three annotation algorithm models include YOLOv8 (hereinafter referred to as Model A), Faster R-CNN (hereinafter referred to as Model B), and DETR (hereinafter referred to as Model C). The test image contains clear vehicles, medium-sized pedestrians, blurred cyclists, and distant traffic signs.
[0020] More preferably, in an exemplary embodiment, the labeled bounding box is a rectangle with the format [x,y,width,height], where x and y represent the coordinates of the top-left vertex of the rectangle, width represents the width of the rectangle, and height represents the height of the rectangle; The intersection-union ratio (IoU) of two bounding boxes is the area of their intersection divided by the area of their union. If the intersection-union ratio (IoU) of the bounding boxes of objects detected by different annotation algorithm models is greater than 0.5, they are considered to be the same object.
[0021] Specifically, in an exemplary embodiment, the outputs of the three models are: The output of model A includes: A1: Labeled bounding box, [420,320,180,380], category, vehicle, confidence score, 0.92; A2: Labeled bounding box, [1200,450,100,220], category, pedestrian, confidence score, 0.87; A3: Labeled bounding box, [800,500,80,160], category, bicycle, confidence score, 0.45.
[0022] The output of Model B includes: B1: Labeled bounding box, [415,315,185,385], category, vehicle, confidence level, 0.95; B2: Labeled bounding box, [1195,445,105,225], category, pedestrian, confidence level, 0.93; B3: Labeled bounding box, [150,300,60,100], category, traffic sign, confidence level, 0.78.
[0023] The output of model C includes: C1: Labeled bounding box, [425,325,175,375], category, vehicle, confidence score, 0.88; C2: Labeled bounding box, [1210,460,90,210], category, pedestrian, confidence score, 0.81; C3: Labeled bounding box, [810,510,70,150], category, bicycle, confidence score, 0.68; C4: Labeled bounding box, [155,305,55,95], category, traffic sign, confidence score, 0.72.
[0024] At this point, for any two bounding boxes, the coordinates of the first bounding box box1 are [x1, y1, w1, h1], and the coordinates of the first bounding box box2 are [x2, y2, w2, h2]. The specific implementation of IoU calculation includes: (1) Convert the coordinates to corner point representation: The top left corner of the first bounding box box1 is [x1, y1], and the bottom right corner of the first bounding box box1 is [x1+w1, y1+h1]; the top left corner of the second bounding box box1 is [x2, y2], and the top right corner of the second bounding box box1 is [x2+w2, y2+h2]; (2) Calculate the intersection rectangle The coordinates of the top left corner of the intersection rectangle are [max(x1,x2), max(y1,y2)], and the coordinates of the top right corner of the intersection rectangle are [min(x1+w1,x2+w2), min(y1+h1,y2+h2)].
[0025] The intersection area is calculated as follows: width inter_width=max(0,min(x1+w1,x2+w2)-max(x1,x2)), height inter_height=max(0,min(y1+h1,y2+h2)-max(y1,y2)), and intersection area inter_area=inter_width*inter_height.
[0026] (3) Calculate the area of each. The area of the first bounding box (box1) is area1 = w1 * h1; the area of the second bounding box (box2) is area2 = w2 * h2.
[0027] (4) Calculate the area of the union The area of the union is equal to the area of area1 plus the area of area2, and the area of the intersection is equal to the area of intersection.
[0028] (5) Calculate IoU IoU = Intersection area / Union area. It should be noted that if the union area is not greater than 0, then IoU is set to 0.
[0029] Based on the above approach, the matching process in this exemplary embodiment is as follows: Group 1: Vehicles (detected by all 3 algorithms). At this point, a three-way matching of A1-B1-C1 is required. IoU(A1,B1)=0.960, IoU(A1,C1)=0.944, IoU(B1,C1)=0.941, and the average IoU within the group is 0.948.
[0030] Group 2: Pedestrians (detected by all 3 algorithms). In this case, a three-way matching of A2-B2-C2 is required. IoU(A2,B2)=0.931, IoU(A2,C2)=0.905, IoU(B2,C2)=0.913, and the average IoU within the group is 0.916.
[0031] Group 3: Bicycle (detected by models A and C, not detected by model B). Only bidirectional matching between A3 and C3 is performed, with IoU(A3,C3)=0.635. Model B did not detect this object; matching quality: medium.
[0032] Group 4: Traffic sign (Detected by models B and C, not detected by model A, only performed in this case) B3-C4 bidirectional matching, IoU(B3,C4)=0.723. Model A did not detect this object, matching quality: good.
[0033] More preferably, in an exemplary embodiment, the calculation of the similarity S between the outputs of the pairwise labeling algorithm model includes: In the formula, Ai and Aj represent the i-th and j-th annotation algorithm models, respectively, and the average matching quality represents the average intersection-union ratio (IoU) of the matched objects in the i-th and j-th annotation algorithm models.
[0034] Specifically, in this exemplary embodiment, the pairwise matching after execution is as follows: 1. Matching of Model A and Model B, matching pairs: 2 (vehicle, pedestrian), average IoU: (0.960+0.931) / 2=0.9455, similarity S(A,B): 2. Matching of Model A and Model C, matching pairs: 3 (vehicle, pedestrian, bicycle), average IoU: (0.944+0.905+0.635) / 3=0.828, similarity S(A,C): 3. Matching of Model B and Model C, matching pairs: 3 (vehicle, pedestrian, sign), average IoU: (0.941+0.913+0.723) / 3=0.859, similarity S(B,C): More preferably, in an exemplary embodiment, obtaining the consensus weight by averaging the similarity with other annotation algorithm models includes: Calculate the consensus average similarity between each annotation algorithm model and other annotation algorithm models; The consensus weight is obtained by normalizing the average value of all consensus values.
[0035] Specifically, in this exemplary embodiment, the consensus average value (W) of the similarity between each annotation algorithm model and other annotation algorithm models is used. A W B W C As described below: The consensus weights are obtained through normalization. First, the sum of the weights is calculated as 0.6255 + 0.6370 + 0.6325 = 1.8950, and then normalization is performed. As can be seen, the weight distribution is very balanced because the three algorithms perform with high consistency on the main objects.
[0036] More preferably, in an exemplary embodiment, the step of performing weighted summation on each object based on the consensus weight to obtain the final annotation result includes: The weighted bounding box is obtained by weighting the bounding box coordinates of each object in each annotation algorithm model with the consensus weight. The weighted bounding box is obtained by weighting the confidence score and consensus weight of each object in each annotation algorithm model. The weighted bounding box and the weighted bounding box are used as the final annotation results.
[0037] Specifically, in this exemplary embodiment, the result after execution is as follows: Group 1: Vehicles, detected consistently by all algorithms: Weighted confidence level: 0.92×0.330+0.95×0.336+0.88×0.334=0.916 Weighted bounding box: x=420×0.330+415×0.336+425×0.334=420; y=320×0.330+315×0.336+325×0.334=320; width=180×0.330+185×0.336+175×0.334=180; height=380×0.330+385×0.336+375×0.334=380; Decision: Bounding box [420,320,180,380], can be automatically labeled with high confidence (0.916).
[0038] Group 2: Pedestrians, consistently detected by all algorithms. Weighted confidence level: 0.87×0.330+0.93×0.336+0.81×0.334=0.868 Weighted bounding box: x=1200×0.330+1195×0.336+1210×0.334=1202; y=450×0.330+445×0.336+460×0.334=452; width=100×0.330+105×0.336+90×0.334=98; height=220×0.330+225×0.336+210×0.334=218; Decision: Bounding box [1202,452,98,218] can be automatically labeled with a confidence level of 0.868.
[0039] Group 3: Bicycles, only models A and C detected them, model B did not detect them. Weighted confidence level (B is considered 0): 0.45 × 0.330 + 0 × 0.336 + 0.68 × 0.334 = 0.376 Weighted confidence score (effective weights renormalized: considering only models A and C, their effective weights in this group are: total effective weights = W_A' + W_C' = 0.330 + 0.334 = 0.664, effective weights of model A = 0.330 / 0.664 = 0.497, effective weights of model C = 0.334 / 0.664 = 0.503): x=800×0.497+810×0.503=805; y=500×0.497+510×0.503=505; width=80×0.497+70×0.503=75; height=160×0.497+150×0.503; IoU quality: 0.635 (Medium) Decision: Bounding box [805,505,75,155] needs to be manually reviewed for the following reasons: (1) an algorithm failed to detect it at all; (2) the weighted confidence level is lower than the threshold (0.5); (3) the detection pair IoU is moderate (0.635).
[0040] Group 4: Traffic signs, only models B and C detected them, model A did not detect them. Weighted confidence level: 0 × 0.330 + 0.78 × 0.336 + 0.72 × 0.334 = 0.502 Weighted confidence score (effective weights renormalized: considering only models B and C, their effective weights in this group are: total effective weight = W_B' + W_C' = 0.336 + 0.334 = 0.670, effective weight of model B = 0.336 / 0.670 = 0.501, effective weight of model C = 0.334 / 0.670 = 0.499): x=150×0.501+155×0.499=152; y=300×0.501+305×0.499=302; width=60×0.501+55×0.499=58; height=100×0.501+95×0.499=98; IoU quality: 0.723 (Good) Decision: Bounding box [152,302,58,98], automatically labeled (edge passed), reason: (1) the two detection algorithms are highly consistent (IoU=0.723); (2) the weighted confidence score just exceeds the threshold (0.502); (3) the marker may actually exist and A was missed. Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.
Claims
1. A consensus labeling decision method based on multiple algorithms, characterized in that: Includes the following steps: Acquire images captured by the vehicle's onboard camera; The image is input into three annotation algorithm models to obtain output results including annotation bounding boxes, annotation categories, and annotation confidence scores; Using the intersection-union ratio (IoU) of the labeled bounding boxes as the matching rule, the same object detected by different labeling algorithm models is matched; Calculate the similarity between the outputs of the pairwise labeling algorithm models, and obtain the consensus weight by using the average similarity with other labeling algorithm models; The final annotation result is obtained by performing weighted calculation on each object based on the consensus weight.
2. The consensus labeling decision method based on multiple algorithms according to claim 1, characterized in that: The labeling categories include pedestrians, vehicles, bicycles, and traffic signs.
3. The consensus labeling decision method based on multiple algorithms according to claim 1, characterized in that: The three annotation algorithm models include YOLOv8, Faster R-CNN, and DETR.
4. The consensus labeling decision method based on multiple algorithms according to claim 1, characterized in that: The bounding box is a rectangle with the format [x,y,width,height], where x and y represent the coordinates of the top-left corner of the rectangle, width represents the width of the rectangle, and height represents the height of the rectangle. The intersection-union ratio (IoU) of two bounding boxes is the area of their intersection divided by the area of their union. If the intersection-union ratio (IoU) of the bounding boxes of objects detected by different annotation algorithm models is greater than 0.5, they are considered to be the same object.
5. The consensus labeling decision method based on multiple algorithms according to claim 4, characterized in that: The similarity S between the outputs of the pairwise labeling algorithm model is calculated using the following methods: In the formula, A i and A j Let represent the i-th and j-th annotation algorithm models, respectively. The average matching quality represents the average intersection-union ratio (IoU) of the matched objects in the i-th and j-th annotation algorithm models.
6. The consensus labeling decision method based on multiple algorithms according to claim 5, characterized in that: The consensus weight is obtained by averaging the similarity with other annotation algorithm models, including: Calculate the consensus average similarity between each annotation algorithm model and other annotation algorithm models; The consensus weight is obtained by normalizing the average value of all consensus values.
7. The consensus labeling decision method based on multiple algorithms according to claim 6, characterized in that: The step of performing weighted summation on each object based on the consensus weight to obtain the final annotation result includes: The weighted bounding box is obtained by weighting the bounding box coordinates of each object in each annotation algorithm model with the consensus weight. The weighted bounding box is obtained by weighting the confidence score and consensus weight of each object in each annotation algorithm model. The weighted bounding box and the weighted bounding box are used as the final annotation results.