A driving data labeling optimization method, device and equipment

Abstract

The application discloses a driving data labeling optimization method, device and equipment, the method comprising: inputting a preset labeling task instruction and a plurality of driving picture data into a pre-labeling model to obtain pre-labeling data corresponding to each driving picture data; for each driving picture data, inputting a preset judging task instruction, the driving picture data and the pre-labeling data corresponding thereto into a pre-judging model to obtain a labeling quality score corresponding to the pre-labeling data; the preset judging task instruction comprises a multi-dimensional scoring rule and a set of veto conditions; based on the labeling quality score, all the pre-labeling data is divided into a plurality of data sets of different quality levels; based on the sampling proportion corresponding to each quality level data set, each data set is respectively subjected to quality inspection. The method can significantly reduce the labor and time cost required for quality inspection under the premise of ensuring the quality of the final labeling data, and improve the overall data labeling efficiency.

Description

Technical Field

[0001] This invention relates to the field of driving data processing technology, and in particular to a driving data annotation and optimization method, apparatus and equipment. Background Technology

[0002] With the rapid development of artificial intelligence technology, visual language models (VLMs) are increasingly being used in fields such as autonomous driving. High-quality labeled data is a crucial foundation for the training and optimization of visual language models. Currently, the mainstream data labeling model in the industry is to pre-label driving scene images using visual language models, followed by manual verification and correction of the pre-labeled results. However, this model still has many shortcomings in practical applications.

[0003] First, the quality of pre-labeled results varies considerably. Limited by model capabilities and scene complexity, the pre-labeling quality of visual language models differs significantly across different driving scenarios. Second, existing quality control strategies are relatively simplistic. When dealing with large volumes of pre-labeled data, manual quality control typically employs fixed-ratio sampling or random sampling, failing to adequately consider the quality differences between different pre-labeled data. This results in a large amount of quality control resources being allocated to already relatively accurate pre-labeled data, while pre-labeled data with significant errors may remain untouched due to sampling. This "one-size-fits-all" approach wastes human resources and leaves potential quality issues unaddressed. Furthermore, balancing accuracy and efficiency is difficult. Increasing the sampling ratio to ensure data accuracy significantly increases the workload of manual quality control and reduces efficiency; conversely, reducing the sampling ratio to pursue efficiency compromises data accuracy. Summary of the Invention

[0004] In order to improve quality inspection efficiency while ensuring the accuracy of labeling, this invention provides a driving data labeling optimization method, device and equipment.

[0005] In a first aspect, embodiments of the present invention provide a driving data annotation optimization method, which may include: Input the preset annotation task instructions and multiple driving image data into the pre-annotation model to obtain the pre-annotation data corresponding to each driving image data; For each driving image data, the preset referee task instruction, the driving image data, and the corresponding pre-annotated data are input into the pre-referee model to obtain the annotation quality score corresponding to the pre-annotated data; the preset referee task instruction includes multi-dimensional scoring rules and a set of veto conditions; Based on the labeled quality scores, all the pre-labeled data are divided into multiple datasets with different quality levels; Quality checks were performed on each dataset based on the sampling ratio corresponding to each quality level.

[0006] In one or more optional embodiments of this application, the preset referee task instruction is obtained in the following manner: Multiple veto conditions are obtained; the veto conditions are error labeling types that are likely to lead to safety accidents. Based on the preset veto for low scores and all the veto conditions, the veto condition set is obtained; The set of veto conditions is added to the preset referee task instruction so that if the pre-referee model identifies any of the veto conditions in the pre-labeled data based on the driving image data, it outputs the labeling quality score as the preset veto low score.

[0007] In one or more optional embodiments of this application, the multi-dimensional scoring rules are obtained in the following manner: The system obtains tiered deduction rules, multiple evaluation dimensions, and the weights corresponding to each evaluation dimension; the tiered deduction rules include multiple error severity levels and corresponding deduction values. The multi-dimensional scoring rule is composed of the tiered deduction rules, multiple evaluation dimensions, and the weights corresponding to each evaluation dimension.

[0008] In one or more optional embodiments of this application, the multiple evaluation dimensions include a traffic control device identification quality evaluation dimension, a key traffic participant identification quality evaluation dimension, a decision reasoning quality evaluation dimension, a format and logic consistency evaluation dimension, and a professional compliance evaluation dimension.

[0009] In one or more optional embodiments of this application, the following further includes: Obtain multiple error annotation records obtained during historical quality inspection processes; At least one high-frequency error labeling type is determined based on the multiple error labeling records; High-risk special attention rules are constructed based on the aforementioned high-frequency error labeling types; Add the high-risk special attention rule to the preset referee task instruction; The high-frequency error labeling type is added to the multi-dimensional scoring rule.

[0010] In one or more optional embodiments of this application, the step of dividing all the pre-labeled data into multiple datasets of different quality levels based on the labeled quality score includes: All the pre-annotated data with quality scores greater than or equal to a preset score threshold are combined into a high-scoring dataset representing a relatively high quality level. All the pre-labeled data with quality scores lower than the preset score threshold are grouped into a low-score dataset representing a relatively low quality level.

[0011] In one or more optional embodiments of this application, the quality inspection of each dataset based on the sampling ratio corresponding to each quality level includes: The high-scoring dataset is quality inspected based on the first sampling ratio; The low-scoring dataset is quality inspected based on a second sampling ratio; the first sampling ratio is less than the second sampling ratio.

[0012] In one or more optional embodiments of this application, the step of inputting a preset annotation task instruction and multiple driving image data into a pre-annotation model to obtain pre-annotated data corresponding to each driving image data includes: For each driving image data, the driving image data and the preset annotation task instruction are input into the pre-annotation model to obtain the pre-annotation data corresponding to the driving image data; the pre-annotation data includes traffic light information, key target objects, detection boxes, and driving decisions.

[0013] Secondly, embodiments of the present invention provide a driving data annotation and optimization device, which may include: The pre-annotation module is used to input a preset annotation task instruction and multiple driving image data into the pre-annotation model to obtain pre-annotation data corresponding to each driving image data. The referee module is used to input the preset referee task instruction, the driving image data and its corresponding pre-annotated data into the pre-referee model for each driving image data, and obtain the annotation quality score corresponding to the pre-annotated data; the preset referee task instruction includes multi-dimensional scoring rules and a set of veto conditions; The partitioning module is used to divide all the pre-labeled data into multiple datasets with different quality levels based on the labeled quality scores; The quality inspection module is used to perform quality inspection on each dataset based on the sampling ratio corresponding to each quality level.

[0014] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the driving data annotation optimization method as described above.

[0015] Fourthly, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the driving data annotation optimization method as described above.

[0016] Fifthly, embodiments of the present invention provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the driving data annotation optimization method as described above.

[0017] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following: This invention provides a method for optimizing driving data annotation. The method first performs preliminary annotation on driving image data using a pre-annotation model. Then, a preset referee task instruction, along with the driving image data and its corresponding pre-annotated data, is input into the pre-referee model. The pre-referee model outputs a standardized annotation quality score. The preset referee task instruction includes multi-dimensional scoring rules and a veto condition set. The veto condition set can directly intercept pre-annotated results with serious safety risks, while the multi-dimensional scoring rules provide a fine-grained quantitative evaluation of the pre-annotation quality from multiple perspectives. The two work together to ensure the comprehensiveness and security of the evaluation results.

[0018] Furthermore, based on the annotation quality score, the pre-annotated data is divided into multiple datasets of different quality levels, and a differentiated sampling ratio is used for quality inspection for different levels. This tilts quality inspection resources toward low-quality data, avoids unnecessary repeated verification of high-quality data, and effectively reduces redundant quality inspection workload. This method can significantly reduce the manpower and time costs required for quality inspection while ensuring the quality of the final annotated data, and improve the overall data annotation efficiency.

[0019] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating the driving data annotation and optimization method provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the driving data annotation and optimization device provided in an embodiment of the present invention. Detailed Implementation

[0022] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0023] The inventors discovered that, in existing technologies, the mainstream data annotation method has become the use of visual language models to pre-annotate driving scene images, followed by manual verification and correction of the pre-annotation results. However, this method still has many shortcomings in practical applications.

[0024] First, the quality of pre-annotation results varies considerably. Limited by model capabilities and scene complexity, the pre-annotation quality of visual language models differs significantly across different driving scenarios. Second, existing quality inspection strategies are relatively simplistic. When dealing with large volumes of pre-annotated data, manual quality inspection typically employs fixed-ratio sampling or random sampling, failing to adequately consider the quality differences between different pre-annotated data. This results in a large amount of quality inspection resources being allocated to already relatively accurate pre-annotated data, while pre-annotated data with significant errors may remain untouched due to sampling. This "one-size-fits-all" approach wastes human resources and leaves potential quality issues unaddressed. Furthermore, balancing accuracy and efficiency is difficult. Increasing the sampling ratio to ensure data accuracy significantly increases the workload of manual quality inspection and reduces efficiency; conversely, decreasing the sampling ratio to pursue efficiency compromises data accuracy. Based on these issues, the inventors, through further research and development, have created this invention, providing a driving data annotation optimization method, apparatus, and device.

[0025] Example 1 Embodiment 1 of the present invention provides a method for optimizing driving data annotation, referring to... Figure 1 As shown, the method may include the following steps S101-S104: S101: Input the preset annotation task instructions and multiple driving image data into the pre-annotation model to obtain the pre-annotation data corresponding to each driving image data.

[0026] S102: For each driving image data set, input the preset referee task instructions, the driving image data, and its corresponding pre-annotated data into the pre-referee model to obtain the annotation quality score corresponding to the pre-annotated data. The preset referee task instructions include multi-dimensional scoring rules and a set of veto conditions.

[0027] S103: Based on the annotation quality score, all pre-labeled data are divided into multiple datasets with different quality levels.

[0028] S104: Based on the sampling ratio corresponding to each quality level of the dataset, perform quality checks on each dataset separately.

[0029] This invention provides a method for optimizing driving data annotation. The method first performs preliminary annotation on driving image data using a pre-annotation model. Then, a pre-defined referee task instruction, along with the driving image data and its corresponding pre-annotated data, is input into the pre-referee model. The pre-referee model outputs a standardized annotation quality score. The pre-defined referee task instruction includes multi-dimensional scoring rules and a set of veto conditions. The veto conditions directly intercept pre-annotated results with serious safety risks, while the multi-dimensional scoring rules provide fine-grained quantitative evaluation of the pre-annotation quality from multiple perspectives. Together, they ensure the comprehensiveness and security of the evaluation results. Furthermore, based on the annotation quality score, the pre-annotated data is divided into multiple datasets of different quality levels, and a differentiated sampling ratio is used for quality inspection of different levels. This tilts quality inspection resources towards lower-quality data, avoiding unnecessary repeated verification of high-quality data and effectively reducing redundant quality inspection workload. This method can significantly reduce the manpower and time costs required for quality inspection while ensuring the quality of the final annotated data, thereby improving the overall data annotation efficiency.

[0030] In step S101 above, the preset annotation task instruction and multiple driving image data are input into the pre-annotation model to obtain the pre-annotation data corresponding to each driving image data.

[0031] Specifically, for each driving image data set, the driving image data and preset annotation task instructions are input into a pre-annotation model. The pre-annotation model then performs a visual question answering (VQA) task on the driving image data to obtain the pre-annotated data corresponding to the driving image data. The pre-annotated data includes traffic light information, key targets, detection boxes, and driving decisions.

[0032] The pre-annotated model can be implemented using a Vision Language Model (VLM), such as the Qwen-VL series, LLaVA series, etc.

[0033] In addition, to improve the understanding ability of the pre-labeled model in complex driving scenarios, this method introduces the Chain of Thought (CoT) technology in the pre-labeling process. This guides the pre-labeled model to perform step-by-step reasoning before outputting the final labeling results. For example, it first identifies traffic light information in the image, then detects key objects and their positions, and finally generates driving decision suggestions based on the perception results, thereby improving the logic and accuracy of the pre-labeling results.

[0034] Pre-labeled data is output in a structured format, such as JSON, to facilitate parsing and processing in subsequent stages. Pre-labeled data includes, but is not limited to, the following: traffic light information (such as traffic light type, color status, arrow direction, numerical value, etc.), key targets and their corresponding 2D bounding boxes (such as the location and category of traffic participants such as vehicles, pedestrians, and two-wheeled vehicles), and driving decisions (such as acceleration, deceleration, stopping, turning, etc.). In practical applications, based on historical statistical data, the proportion of truly high-quality data generated by the pre-labeling model is approximately 35%, meaning that most pre-labeled data still contains errors or omissions to varying degrees. Quality assessment and manual quality checks are required in subsequent steps to ensure the accuracy and usability of the final labeled data.

[0035] In step S102 above, for each driving image data, the preset referee task instruction, the driving image data, and its corresponding pre-annotated data are input into the pre-referee model to obtain the annotation quality score corresponding to the pre-annotated data. The preset referee task instruction includes multi-dimensional scoring rules and a set of veto conditions.

[0036] Specifically, for each driving image data, the preset referee task instruction, along with the driving image data and its corresponding pre-annotated data, is input into the pre-referee model. The pre-referee model then evaluates the content of the pre-annotated data according to the preset referee task instruction and outputs the annotation quality score corresponding to the pre-annotated data.

[0037] The pre-judgment model can be implemented using a large language model, such as Qwen3-235B-A22B or Qwen3-VL-32B-Instruct-AWQ.

[0038] The pre-defined referee task instruction includes evaluation logic customized for driving scenarios. This evaluation logic comprises two parts: multi-dimensional scoring rules and a set of veto conditions. The veto condition set defines several error types directly related to safety, used for preliminary screening of the pre-labeled data. If the pre-referee model identifies any of these veto conditions in the pre-labeled data, it directly outputs a labeling quality score below a preset score threshold, and subsequent multi-dimensional scoring is not performed, as detailed in steps S1021-S1023 below. If no veto condition is triggered, the pre-labeled data is scored from multiple angles using fine-grained methods according to the multi-dimensional scoring rules.

[0039] In this way, on the one hand, a veto mechanism is used to block pre-labeled data with serious security risks, and on the other hand, multi-dimensional scoring is used to achieve a comprehensive quantitative assessment of the quality of pre-labeled data and output a standardized labeling quality score.

[0040] The preset referee task instruction is obtained through the following method, specifically including the following steps S1021-S1023: S1021: Obtain multiple veto conditions. Veto conditions are error labeling types that are likely to lead to safety incidents.

[0041] Specifically, this could involve pre-configuring multiple veto conditions, each corresponding to a type of erroneous labeling that is likely to lead to safety accidents. These veto conditions could be derived from inductive analysis of erroneous labels discovered during historical quality inspections. For example, certain types of errors discovered during manual quality inspections (such as incorrect identification of traffic light status or missed detection of pedestrians) would directly result in serious safety hazards in driving decisions made based on the pre-labeled data.

[0042] Therefore, setting this type of erroneous labeling as a veto condition is used for pre-screening of pre-labeled data.

[0043] For example, veto conditions may include the following types: misidentifying a red traffic light as a green traffic light and suggesting passage accordingly, failing to detect a pedestrian crossing the road, misinterpreting a traffic sign as a traffic light and generating a driving decision based on this misinterpretation, and failing to detect a key target, resulting in a lack of avoidance decision-making. The above examples are merely illustrative of veto conditions; in actual applications, more or fewer veto conditions can be configured according to specific driving scenarios and safety requirements.

[0044] S1022: Based on the preset veto for low scores and all veto conditions, obtain the veto condition set.

[0045] Specifically, after obtaining all veto conditions, the veto conditions can be combined into a veto condition set, and a preset low veto score can be associated with this condition set. The preset low veto score is a pre-set fixed score that represents the quality score that the pre-labeled data should obtain when it triggers any of the veto conditions.

[0046] For example, the preset veto low score can be set to 0 points, or it can be set to other low scores in the range of 0 to 100 points (such as 10 points, 20 points, etc.) according to the actual evaluation range. This application embodiment does not specifically limit this.

[0047] By associating the preset low score for veto with the set of veto conditions, the set of veto conditions can be added to the preset referee task instructions, so that when the veto conditions are triggered, the preset low score for veto can be output directly without going through the item-by-item scoring of the multi-dimensional scoring rules.

[0048] S1023: Add the set of veto conditions to the preset referee task instructions so that if the pre-referee model identifies any veto condition in the pre-labeled data based on the driving image data, the output label quality score is the preset veto low score.

[0049] Specifically, the set of veto conditions and their associated preset low veto scores can be included as part of a preset referee task instruction, and input into the pre-referee model along with the driving image data and pre-annotated data in step S102. Upon receiving the preset referee task instruction, the pre-referee model first checks each condition in the veto condition set during quality assessment, that is, it determines whether any erroneous annotation type corresponding to a veto condition exists in the pre-annotated data based on the driving image data. If the pre-referee model identifies any of the aforementioned veto conditions in the pre-annotated data, it directly outputs the annotation quality score as the preset low veto score and does not execute subsequent multi-dimensional scoring rules; if no veto condition is triggered, the multi-dimensional scoring rules continue to be executed.

[0050] Through the above methods, the veto mechanism has the highest priority in the evaluation process, ensuring that pre-labeled data with serious security risks are directly marked as low-scoring, thus avoiding the omission of security risks.

[0051] The multi-dimensional scoring rules are obtained through the following methods, specifically including the following steps S1024-S1025: S1024: Obtain the tiered penalty rules, multiple assessment dimensions, and the corresponding weights for each assessment dimension. The tiered penalty rules include multiple error severity levels and their corresponding penalty values. The multiple assessment dimensions include the traffic control device identification quality assessment dimension, the key traffic participant identification quality assessment dimension, the decision-making reasoning quality assessment dimension, the format and logical consistency assessment dimension, and the professional compliance assessment dimension.

[0052] Specifically, this can involve pre-configuring multiple evaluation dimensions, the corresponding weights for each dimension, and the tiered deduction rules for each dimension. The evaluation dimensions reflect different perspectives used to assess the quality of the pre-labeled data.

[0053] The evaluation dimensions include: Traffic control equipment identification quality evaluation dimension, used to evaluate the accuracy of the existence judgment, type recognition, color and value recognition, and lane association logic of traffic lights (such as round lights, arrow lights, and digital lights) in the pre-labeled data; Key traffic participant identification quality evaluation dimension, used to evaluate the completeness of detection, classification accuracy, and rationality of behavioral intention prediction of traffic participants such as vehicles, pedestrians, and two-wheeled vehicles in the pre-labeled data; Decision reasoning quality evaluation dimension, used to evaluate whether the driving decisions generated based on perception results in the pre-labeled data are accurate and reasonable, and whether there are link vulnerabilities caused by front-end perception errors leading to decision deviations; Format and logic consistency evaluation dimension, used to evaluate the format standardization, field completeness, and logical self-consistency between modules of the pre-labeled data; Professional compliance evaluation dimension, used to evaluate whether the pre-labeled data complies with traffic regulations and defensive driving concepts.

[0054] Each assessment dimension has a corresponding weight, which reflects the relative importance of each dimension in the quality score. Since traffic control device recognition errors (such as misinterpreting a red light as a green light) and missed detections of key traffic participants (such as failing to identify pedestrians crossing the road) directly affect driving safety, the traffic control device recognition quality assessment dimension and the key traffic participant recognition quality assessment dimension can be assigned higher weights; while the format and logical consistency assessment dimension does not directly affect safety and can be assigned a lower weight. For example, the weight corresponding to the traffic control device recognition quality assessment dimension is 40%, the key traffic participant recognition quality assessment dimension is 25%, the decision-making reasoning quality assessment dimension is 20%, the format and logical consistency assessment dimension is 10%, and the professional compliance assessment dimension is 5%.

[0055] The tiered deduction rule is a uniform deduction rule applicable to all evaluation dimensions, used to differentiate deductions for errors of varying severity. For example, the tiered deduction rule includes multiple error severity levels and corresponding deduction values. The error severity levels can be divided into fatal errors, serious errors, minor errors, and minor flaws. Fatal errors are safety risk errors that could lead to accidents, with a deduction of 20 to 30 points per instance; serious errors are core errors affecting functional correctness, with a deduction of 10 to 20 points per instance; minor errors are errors that do not affect safety but reduce user experience, with a deduction of 5 to 10 points per instance; and minor flaws are formatting or expression issues, with a deduction of 1 to 5 points per instance.

[0056] By linking the severity level of an error with the deduction value, the pre-judgment model can differentiate the deduction based on the severity of the error during scoring, achieving a more granular quality assessment. Moreover, this deduction rule applies uniformly to all assessment dimensions, meaning that regardless of which assessment dimension the error occurs in, the corresponding deduction value is determined according to the same error severity standard.

[0057] S1025: A multi-dimensional scoring rule is formed based on a tiered deduction rule, multiple evaluation dimensions, and the weights corresponding to each evaluation dimension.

[0058] Specifically, the multiple evaluation dimensions obtained in step S1024, the weight coefficients corresponding to each evaluation dimension, and the tiered deduction rules can be combined to form a multi-dimensional scoring rule. This multi-dimensional scoring rule, as a component of the preset referee task instruction, is input into the pre-referee model along with the driving image data and pre-labeled data in step S102 to guide the pre-referee model in performing routine quality assessments on the pre-labeled data that has not triggered the veto condition.

[0059] In the multi-dimensional scoring rule, the pre-judgment model detects each error in each evaluation dimension and determines the severity level and deduction value of each error according to a unified tiered deduction rule, thereby calculating the dimension score for that evaluation dimension. After scoring all evaluation dimensions, the scores of each dimension are weighted and summed according to the weight coefficients corresponding to each evaluation dimension to obtain the final annotation quality score.

[0060] In this embodiment of the application, the multi-dimensional scoring rules are also updated based on the error labeling records in the historical quality inspection process, specifically including the following steps S10201-S10205: S10201: Obtain multiple error annotation records obtained during historical quality inspection processes.

[0061] Specifically, this could involve collecting and recording error annotation records from historical manual quality inspection processes, resulting from corrections made to pre-annotated data. These error annotation records reflect the specific types and distribution of errors that the pre-annotated model is prone to make in different driving scenarios. For example, during manual quality inspection, after quality inspectors correct incorrectly identified traffic lights, missed pedestrians, and misclassified objects in the pre-annotated data, the corresponding error information (including error type, error location, correction result, etc.) is recorded and stored in a database, forming a historical dataset of error annotation records.

[0062] S10202: Determine at least one high-frequency error labeling type based on multiple error labeling records.

[0063] Specifically, it can be done by statistically analyzing the multiple error annotation records obtained in step S10201, counting the frequency of each error annotation type in the historical quality inspection process, and thus determining at least one error annotation type with a high frequency of occurrence.

[0064] For example, statistics can reveal that error types such as "incorrect recognition of traffic light color", "failure to detect pedestrians", and "misjudging traffic signs as traffic lights" occur frequently, and these can be identified as high-frequency error labeling types.

[0065] The identification of high-frequency error annotation types reflects the systematic defects of the pre-annotated model in specific scenarios, providing a data basis for subsequent targeted optimization of the pre-set referee task instructions.

[0066] S10203: Construct high-risk special attention rules based on high-frequency error annotation types.

[0067] Specifically, based on the high-frequency error labeling types determined in step S10202, a high-risk special attention rule corresponding to the high-frequency error labeling type can be constructed. The high-risk special attention rule is used to instruct the pre-judgment model to assign higher detection sensitivity to the evaluation content related to the high-frequency error labeling type when evaluating the pre-labeled data.

[0068] For example, if the high-frequency error labeling type is "traffic light color recognition error" or "traffic light missed detection", then the corresponding high-risk special attention rule indication pre-judgment model focuses on the traffic light recognition quality, including the accuracy of the existence judgment, type recognition, color and value recognition, and lane association logic of the traffic light; if the high-frequency error labeling type is "pedestrian missed detection" or "target classification error", then the corresponding high-risk special attention rule indication pre-judgment model focuses on the recognition quality of key traffic participants, including the completeness of detection of occluded or unconventional target shapes, the accuracy of target classification, and the rationality of behavioral intention prediction.

[0069] S10204: Add the high-risk special attention rule to the preset referee task instruction.

[0070] Specifically, the high-risk special attention rule constructed in step S10203 can be incorporated as part of the preset referee task instruction, serving as another important component distinct from the multi-dimensional scoring rules and the veto condition set. This allows the pre-referee model to perform not only routine evaluations according to the multi-dimensional scoring rules and the veto condition set, but also to conduct focused verification of corresponding evaluation dimensions or content based on the high-risk special attention rule during quality assessment. By incorporating the high-risk special attention rule into the preset referee task instruction, the pre-referee model can perform targeted enhanced evaluations of high-frequency error types discovered in historical quality inspections, improving its ability to identify error-prone scenarios.

[0071] S10205: Add high-frequency error labeling types to the multi-dimensional scoring rules.

[0072] Specifically, similarly, high-frequency error labeling types also need to be included in the multi-dimensional scoring rules, becoming a key focus item. For example, if the high-frequency error labeling type is "traffic light color recognition error" or "traffic light missed detection," it should be added to the traffic control equipment recognition quality assessment dimension in the multi-dimensional scoring rules, and used as a key verification item in that dimension; if the high-frequency error labeling type is "pedestrian missed detection" or "target classification error," it should be added to the key traffic participant recognition quality assessment dimension in the multi-dimensional scoring rules, and used as a key verification item in that dimension; if the high-frequency error labeling type is "decision reasoning error," it should be added to the decision reasoning quality assessment dimension in the multi-dimensional scoring rules, and used as a key verification item in that dimension.

[0073] By incorporating high-frequency error labeling types into the corresponding evaluation dimensions of the multi-dimensional scoring rules, the pre-judgment model becomes more sensitive to high-frequency errors found in historical quality inspections during subsequent evaluation processes, thereby achieving closed-loop optimization of the evaluation rules.

[0074] To facilitate understanding of this method by those skilled in the art, a complete explanation of the preset referee task instructions is provided here with specific examples: Task: Quality assessment of pre-annotation results for driving scenarios (out of 100 points) Background: You are a senior autonomous driving perception algorithm expert responsible for rigorous quality assessment of the understanding results of pre-labeled driving scenarios. Based on systemic defects discovered in previous quality checks, this assessment will focus on known weaknesses.

[0075] ##Key Assessment Areas## **Special attention should be paid to the following high-risk issues:** 1. **Traffic light recognition:** Type misjudgment, missed detection, color / value recognition error, lane association error.

[0076] 2. **Key Target Recognition**: Missed detection of occluded / unconventional target shapes, incorrect target classification, and misjudgment of pedestrian behavior.

[0077] 3. **Vulnerability of the decision-making chain: Decision bias caused by errors in the underlying identification process.

[0078] ##Evaluation Principles## Strictness: A zero-tolerance standard is adopted for known weaknesses, and any error will result in significant point deductions.

[0079] Risk-oriented: Safety hazards are subject to veto and must be clearly marked and displayed.

[0080] Chain of evidence verification: requires that each judgment be supported by clear visual evidence.

[0081] Systems thinking: Assess the integrity of the "identification-decision" chain, rather than isolated modules.

[0082] ##Rating Dimensions## 1. Traffic control equipment recognition quality (weight: 40%) **[Core Focus - Historical Accuracy Rate Only 70%]** **Traffic Light Existence Verification:** - **Strictly Prohibit "Creating a Signal Out of Nothing":** Misclassifying non-traffic light objects (taillights, streetlights, advertising screens) as traffic lights. - **Strictly Prohibit "Missing Detection":** Failing to identify clearly visible traffic lights. - **Must Verify Through Structural Features:** Judgment must be based on geometric shape, internal patterns, and projection features, not simply color. **Type Recognition Accuracy:** - Round / Arrow Light Confusion: Internal uniformity and directional patterns not checked. - Arrow Direction Judgment: Not based on sharp vertex geometric pointing. - Digital Light Recognition: Failed 7-segment LED display feature verification. **Color and Numerical Stability:** - Color Recognition: Errors caused by lighting / clarity. - Numerical Recognition: Digital light values ​​not detected or incorrect. **Lane Association Logic:** Traffic light and lane matching error: Overall traffic scenario logic not established. **Vetotal Disqualification:** Recommending to proceed at a red light or to stop at a green light is directly deemed a safety risk. ### 2. Quality of Key Traffic Participant Identification (Weight: 25%) **[Key Focus - 15 Identification Errors]** **Target Detection Completeness:** - Occlusion / unconventional target missed - Static obstacles misidentified as dynamic participants (or vice versa) **Target Classification Accuracy:** - Incorrect Classification of Vehicles / Two-Wheeled Vehicles / Pedestrians - Special Concern: Misclassifying motorcycles as bicycles, or misclassifying stationary pedestrians as moving. **Behavioral Intent Prediction:** - Pedestrian behavior misjudgment: Insistence not based on a comprehensive assessment of body orientation, gait, and line of sight. - Vehicle intent prediction: Intention speculation without visual evidence. - **Key Filtering Logic**: - Errors exclude truly critical targets (such as pedestrians about to cross the road). - Errors include irrelevant targets (such as vehicles in the opposite lane). ### 3. Robustness of decision-making reasoning (weight: 20%) **[Core Focus - Identifying the Chain Reaction of Errors]** **Decision-Based Reliability:** Decisions made based on erroneous perceived information – Decisions made without verifying the reliability of perceived results. **Link Vulnerability Assessment:** - Minor errors at the front end lead to major decision-making errors at the back end. - The impact of identification uncertainties on decision-making was not considered. - **Safety First Principle:** - Failure to adopt a conservative strategy when identifying uncertainty - Recommendation of dangerous actions (such as running red lights, failing to yield to pedestrians) is directly classified as a safety risk. ### 4. Format and logical consistency (weight: 10%) -**JSON formatting specification**: Formatting error **Field Integrity:** Key fields are missing. -**Logical consistency: contradictions between modules** -**Safety Statement**: Precise distance / speed values ​​are displayed. ### 5. Professional compliance (weight: 5%) **Traffic Law Compliance: Recommendations for Violating Basic Traffic Regulations** **Defensive Driving:** Fails to embody the concept of preventative driving. ##A veto item (instantly awarded 0 points) 1. It is recommended to proceed when the light is red or to stop when the light is green. 2. Pedestrians crossing the road were not identified. 3. Misinterpreting traffic signs as traffic lights and making decisions based on this. 4. Missed detection of key objectives leads to inadequate avoidance decision-making. ##Evaluation Process 1. **Safety Screening**: First, check if there are any veto items. 2. **Key Module Audit:** Each dimension is scored in depth according to its weight. 3. **Chain of Evidence Verification**: Tracing visual evidence for each judgment. 4. **Link Analysis:** Assessing the impact of identification errors on decision-making. ##Tiered Deduction Rules **Fatal Error:** Safety risks that could lead to an accident (deduct 20-30 points per instance) **Critical Error:** A core error that affects the correctness of the function (deduct 10-20 points per error). **General Errors:** Errors that do not affect security but degrade the user experience (deduct 5-10 points per error). - **Minor flaws:** Formatting or wording issues (deduct 1-5 points per example) ## Scoring criteria (Total score 0-100, 60 points is the passing grade) -**90-100 points (Excellent)**: The perceived results are completely consistent with the visual facts, no key elements are omitted, and the decision-making logic is perfect.

[0083] **80-89 points (Good):** Perception results are accurate. Minor flaws that do not affect core decisions are permissible (such as minor adjustments to attributes of distant, non-critical objectives).

[0084] **60-79 points (passing):** There are a few minor general errors (such as the traffic light type is correctly identified but the value fluctuates slightly), but the overall decision-making logic is reasonable and there are no safety hazards.

[0085] -**40-59 points (poor): There are misjudgments of core elements (such as missed detection of key pedestrians, misjudgment of traffic light colors), which pose potential safety risks.

[0086] -**0-39 points (Very Poor)**: There is a fatal error (such as running a red light, missing serious avoidance), or the annotation result is completely unusable.

[0087] ##Evaluation Process 1. **Prioritize finding reasons for a high score**: If the core signal lights and key participants are all correct, the starting score should be 85 points.

[0088] 2. **Risk Verification**: Check for any fatal errors that could lead to an accident. If none are found, the score should not be lower than 60 points.

[0089] 3. **Tiered Deduction Points**: Based on the “Tiered Deduction Points Rules”, the starting score is slightly adjusted.

[0090] The above is a specific example of a pre-defined referee task instruction. The "Key Assessment Areas" section corresponds to the aforementioned high-risk special attention rule; the "Assessment Principles" section is used to set the assessment behavior guidelines for the pre-referee model, guiding its decision-making logic when performing multi-dimensional scoring and veto screening; the "Scoring Dimensions" section corresponds to the assessment dimensions and weights in the aforementioned multi-dimensional scoring rules, where the percentage after each dimension is the corresponding weight coefficient, and the specific assessment content within each dimension is the fine-grained scoring basis under each assessment dimension; the "Veto Items" section corresponds to the aforementioned veto condition set; and the "Scoring Segmentation Standards" section is used to set the grading basis for the labeled quality score. The pre-referee model classifies the pre-labeled data into the corresponding quality level based on the final score, facilitating the grading processing in subsequent steps.

[0091] It should be noted that the content and parameter values ​​in the above-mentioned example of preset referee task instructions (including the number of evaluation dimensions, the weight percentage of each dimension, the tiered deduction levels and deduction ranges, the preset score threshold of 60 points, etc.) are merely illustrative examples of this application and do not constitute a limitation on the content of the preset referee task instructions. In practical applications, the above-mentioned content and parameter values ​​can be adaptively adjusted and configured according to different driving scenarios, safety requirements, and the actual performance of the pre-labeled model.

[0092] Based on the aforementioned pre-defined referee task instructions, the pre-referee model evaluates the pre-labeled data according to these instructions and then returns the evaluation results in a pre-defined output format. The output content of the pre-referee model is explained below.

[0093] The pre-judgment model outputs a labeling quality score and evaluation details for each pre-labeled data point in a structured format, such as JSON. The output includes the following fields: The overall_score field contains a comprehensive score, a quality grade, and a summary. The comprehensive score is the total score obtained by weighting and summing according to multi-dimensional scoring rules, which is the labeling quality score. The quality grade is a level divided according to the scoring segmentation standards (such as excellent, good, satisfactory, poor, and very poor). The summary is used to briefly summarize the main problems or overall quality of the pre-labeled data, such as "serious defects exist in traffic light recognition, key target recognition is missed, and the decision-making link is fragile."

[0094] The `dimension_scores` field contains the independent score, error count, critical error list, and risk level for each assessment dimension. The independent score for each assessment dimension reflects the quality level under that dimension, the critical error list records the specific errors identified under that dimension, and the risk level indicates the degree of risk for that dimension (e.g., high, medium, low).

[0095] The `critical_issues` field records triggering veto conditions or critical security issues. When the pre-judgment model identifies a veto condition in the pre-labeled data, it records the specific issue corresponding to that condition in this field. This field is used to quickly locate pre-labeled data with serious security risks.

[0096] The `error_analysis` field contains structured error statistics. For example, it may include statistics on traffic light errors (such as the number of false alarms, false alarms, color classification errors, and lane association errors), participant errors (such as the number of missed detections, classification errors, and behavior prediction errors), and decision-making errors (such as the number of decisions based on error perception and the number of safety violations). This field is used to accumulate error data to support subsequent closed-loop optimization.

[0097] The `improvement_recommendations` field records the improvement suggestions generated by the pre-judgment model for problems identified in the current pre-labeled data. Examples include suggestions such as "strengthen traffic light structural feature recognition," "improve occluded target detection algorithms," and "establish a perception-decision verification mechanism." The suggestions output in this field can serve as a reference for subsequent optimization of the pre-labeled model or evaluation rules.

[0098] The `assessment_metadata` field contains metadata information about the assessment. For example, it may include the assessor identifier (e.g., the model or version of the pre-assessment model), timestamp, image path, risk assessment level, and whether it was based on a quality control report (based on a QA report). This field is used to record traceability information for the assessment, facilitating subsequent auditing and analysis.

[0099] Through the structured outputs described above, the pre-judgment model not only provides standardized annotation quality scores, but also detailed scores for each dimension, records of key issues, error statistical analysis, improvement suggestions, and assessment tracing information. These outputs collectively constitute a complete evaluation report, providing a data foundation for subsequent data grading, sampling quality checks, and closed-loop optimization.

[0100] In step S103 above, based on the annotation quality score, all pre-labeled data are divided into multiple datasets with different quality levels. Specifically, this includes the following steps S1031-S1032: S1031: All pre-labeled data with quality scores greater than or equal to a preset score threshold are combined into a high-scoring dataset representing a relatively high quality level.

[0101] S1032: All pre-labeled data with quality scores lower than a preset score threshold are combined into a low-score dataset representing a relatively low quality level.

[0102] Specifically, this can involve obtaining the annotation quality score corresponding to each pre-annotated data output by the pre-judging model in step S102, and comparing each annotation quality score with a preset score threshold. Pre-annotated data with an annotation quality score greater than or equal to the preset score threshold indicates that the pre-judging model considers the pre-annotated data to be of high quality and has a low probability of containing errors, and thus classifies it into the high-scoring dataset. It should be noted that "high score" and "low score" are relative concepts. A high-scoring dataset only indicates that the pre-annotated data in this dataset is judged by the pre-judging model to be of relatively high quality, and does not mean that its annotation is completely error-free.

[0103] For example, the preset score threshold can be set to 60 points. All pre-labeled data with a quality score greater than or equal to 60 points will be classified into the high-scoring dataset, and all pre-labeled data with a quality score less than 60 points will be classified into the low-scoring dataset. The specific value of the preset score threshold can be flexibly adjusted according to the actual application scenario and the requirements for data quality. This application embodiment does not limit this.

[0104] In step S104 above, quality checks are performed on each dataset based on the sampling ratio corresponding to each quality level. Specifically, this includes the following steps S1041-S1042: S1041: Perform quality checks on the high-scoring dataset based on the first sampling ratio.

[0105] S1042: Perform quality checks on the low-scoring dataset based on the second sampling ratio. The first sampling ratio is smaller than the second sampling ratio.

[0106] Specifically, after determining the high-scoring and low-scoring datasets, corresponding sampling ratios are configured for each, with the second sampling ratio for the low-scoring dataset being higher than the first sampling ratio for the high-scoring dataset. During execution, pre-labeled data is extracted from the high-scoring dataset according to the first sampling ratio for manual quality inspection, and pre-labeled data is extracted from the low-scoring dataset according to the second sampling ratio for manual quality inspection.

[0107] Because data in low-scoring datasets are considered to have a higher probability of error, a higher sampling ratio is used to ensure their quality; data in high-scoring datasets are considered to have a lower probability of error, so a lower sampling ratio is used to reduce unnecessary workload. This differentiated sampling strategy effectively reduces the workload of manual quality inspection without significantly lowering the overall accuracy.

[0108] In this step, the specific values ​​of the first and second sampling ratios can be determined comprehensively based on the scoring accuracy of the pre-judgment model, the proportion of true high-quality data in the pre-labeled model, and the desired final accuracy. In one specific implementation, the appropriate sampling ratio can be determined through the following mathematical calculations.

[0109] Let the total data volume be N, the proportion of truly high-quality data in the pre-labeled model be P (i.e., the pass rate), the scoring accuracy of the pre-judging model be A (i.e., the accuracy of the judging model in judging the quality of the pre-labeled data), the sampling ratio of the high-scoring dataset be S1, and the sampling ratio of the low-scoring dataset be S2. Among the above parameters, P can be obtained through historical statistics (e.g., the pass rate of the pre-labeled model is 35% based on historical quality inspection results), A can be obtained by comparing the scoring results of the pre-judging model with the results of human quality inspection (e.g., the scoring accuracy of the pre-judging model is 65%), and S1 and S2 are the sampling ratios to be configured.

[0110] Based on the above parameters, the following mathematical relationship can be established. The number of truly high-quality data is N×P, and the number of truly low-quality data is N×(1-P). The correspondence between the pre-judgment model's scoring results and the true quality can be described by a confusion matrix: the number of truly high-quality data that is rated as high-quality (true positive, TP) is N×P×A; the number of truly high-quality data that is rated as low-quality (false negative, FN) is N×P×(1-A); the number of truly low-quality data that is rated as high-quality (false positive, FP) is N×(1-P)×(1-A); and the number of truly low-quality data that is rated as low-quality (true negative, TN) is N×(1-P)×A. Therefore, the total number of high-quality scores (i.e., the amount of data classified into the high-scoring dataset) is TP+FP, and the total number of low-quality scores (i.e., the amount of data classified into the low-scoring dataset) is FN+TN.

[0111] Furthermore, the sampling size for the high-scoring dataset is (TP+FP)×S1, and the sampling size for the low-scoring dataset is (FN+TN)×S2, with the total manual sampling size being the sum of the two. The final data accuracy consists of two parts: the manually sampled quality control portion (assuming the results after manual quality control are 100% correct) and the inherently correct data in the unsampled portion. By adjusting the values ​​of S1 and S2, a balance can be struck between the total manual sampling size and the final data accuracy.

[0112] For example, given a total data volume N=1000, a pass rate P=35%, and a scoring accuracy A=65%, if the sampling ratio of high-scoring datasets S1=40% and the sampling ratio of low-scoring datasets S2=100%, then the total manual sampling quantity can be calculated using the above mathematical relationship as follows: 727 samples (accounting for 72.7% of the total), and the final data accuracy rate is 86.35%. As can be seen from the above example, the sampling calculation logic in this embodiment can significantly reduce the workload of manual quality inspection while ensuring a high final accuracy rate, requiring only the inspection of 72.7% of all data. In specific implementations, the values ​​of S1 and S2 can be set according to business needs (such as the expected lower limit of the final accuracy rate or the upper limit of available human resources) to achieve an optimal balance between accuracy and efficiency in the quality inspection strategy.

[0113] The embodiments of this application significantly improve quality inspection efficiency while ensuring the accuracy of the final data.

[0114] First, in the pre-annotation stage, by introducing a visual language model and combining it with the thinking chain technology to process driving image data through visual question answering tasks, the model's batch processing capabilities can be fully utilized to complete the initial annotation of large-scale data in a short period of time, providing a data foundation for subsequent quality inspection.

[0115] During the quality assessment phase, a pre-judgment model is automatically evaluated for the pre-annotated results by pre-set referee task instructions. These instructions carry customized assessment logic for driving scenarios, comprising three levels: a veto condition set, multi-dimensional scoring rules, and high-risk special attention rules. The veto condition set directly intercepts and outputs low scores for pre-annotated results with serious safety risks (such as incorrect traffic light status or missed pedestrian detection), preventing fatal errors from being diluted in the regular weighted scoring. The multi-dimensional scoring rules provide fine-grained quantitative assessment of pre-annotation quality from multiple perspectives, including traffic control equipment recognition quality, key traffic participant recognition quality, decision-making reasoning quality, format and logical consistency, and professional compliance. A unified tiered deduction rule differentiates the deduction for errors of varying severity. The high-risk special attention rules provide targeted reinforcement assessment based on frequently occurring error types identified during historical quality inspections. These three levels work together to ensure that safety red lines are not breached while guaranteeing the comprehensiveness and relevance of the assessment results, providing a reliable quantitative basis for subsequent differentiated quality inspections.

[0116] Based on this, pre-labeled data is divided into high-scoring and low-scoring datasets according to the labeling quality score, and corresponding sampling ratios are configured for datasets of different quality levels. This allows quality inspection resources to be tilted towards low-quality data, avoiding unnecessary duplicate verification of high-quality data. Simultaneously, by establishing mathematical relationships based on parameters such as the scoring accuracy of the pre-judgment model and the proportion of truly high-quality data in the pre-labeling model, the expected manual sampling volume and expected final accuracy under different sampling ratio configurations can be accurately calculated, achieving quantifiable, predictable, and optimizable quality inspection strategies. For example, with a scoring accuracy of 65% and a truly high-quality data proportion of 35%, performing differentiated quality inspection with a 40% sampling ratio for the high-scoring dataset and a 100% sampling ratio for the low-scoring dataset achieves a final accuracy of 86.35% with only 72.7% of the manual workload. In practical applications, the sampling ratio can also be optimized in reverse according to business needs, achieving a dynamic balance between accuracy and efficiency in the quality inspection strategy.

[0117] Furthermore, the aforementioned method possesses closed-loop optimization capabilities. By acquiring error annotation records from historical quality inspection processes, high-frequency error annotation types are identified, and updates to high-risk special attention rules and multi-dimensional scoring rules are constructed accordingly. This allows the evaluation rules of the pre-judgment model to continuously iterate and optimize with the accumulation of quality inspection data, forming a virtuous cycle of "evaluation-quality inspection-feedback-update." Moreover, the modules employ a loosely coupled design, facilitating the replacement of pre-annotation or pre-judgment models with different performance levels. It can also be extended to support quality inspection needs for various data types, exhibiting strong scalability and adaptability.

[0118] In summary, this method can significantly reduce the manpower and time costs required for quality inspection while ensuring the quality of the final labeled data, effectively solving the technical problem of balancing quality inspection efficiency and data accuracy in existing technologies.

[0119] Example 2 Based on the same inventive concept, embodiments of the present invention also provide a driving data annotation and optimization device, referring to... Figure 2 As shown, the device includes: The pre-annotation module 101 is used to input a preset annotation task instruction and multiple driving image data into the pre-annotation model to obtain pre-annotation data corresponding to each driving image data. The referee module 102 is used to input a preset referee task instruction, the driving image data and its corresponding pre-annotated data into the pre-referee model for each driving image data, and obtain the annotation quality score corresponding to the pre-annotated data; the preset referee task instruction includes multi-dimensional scoring rules and a set of veto conditions; The partitioning module 103 is used to partition all the pre-labeled data into multiple datasets with different quality levels based on the labeled quality scores; The quality inspection module 104 is used to perform quality inspection on each dataset based on the sampling ratio corresponding to each quality level.

[0120] Example 3 Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program / instruction thereon, which, when executed by a processor, implements the driving data annotation optimization method as described in Embodiment 1 above.

[0121] Example 4 Based on the same inventive concept, embodiments of the present invention also provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the driving data annotation optimization method as described in Embodiment 1 above.

[0122] Example 5 Based on the same inventive concept, this embodiment of the invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the driving data annotation optimization method as described in Embodiment 1 above.

[0123] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0124] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0127] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for optimizing driving data annotation, characterized in that, include: Input the preset annotation task instructions and multiple driving image data into the pre-annotation model to obtain the pre-annotation data corresponding to each driving image data; For each driving image data, the preset referee task instruction, the driving image data and its corresponding pre-annotated data are input into the pre-referee model to obtain the annotation quality score corresponding to the pre-annotated data; The preset referee task instructions include multi-dimensional scoring rules and a set of veto conditions; Based on the labeled quality scores, all the pre-labeled data are divided into multiple datasets with different quality levels; Quality checks were performed on each dataset based on the sampling ratio corresponding to each quality level.

2. The method according to claim 1, characterized in that, The preset referee task instruction is obtained in the following manner: Multiple veto conditions are obtained; the veto conditions are error labeling types that are likely to lead to safety accidents. Based on the preset veto for low scores and all the veto conditions, the veto condition set is obtained; The set of veto conditions is added to the preset referee task instruction so that if the pre-referee model identifies any of the veto conditions in the pre-labeled data based on the driving image data, it outputs the labeling quality score as the preset veto low score.

3. The method according to claim 1, characterized in that, The multi-dimensional scoring rules are obtained in the following manner: The system obtains tiered deduction rules, multiple evaluation dimensions, and the weights corresponding to each evaluation dimension; the tiered deduction rules include multiple error severity levels and corresponding deduction values. The multi-dimensional scoring rule is composed of the tiered deduction rules, multiple evaluation dimensions, and the weights corresponding to each evaluation dimension.

4. The method according to claim 3, characterized in that, The multiple assessment dimensions include the traffic control equipment identification quality assessment dimension, the key traffic participant identification quality assessment dimension, the decision reasoning quality assessment dimension, the format and logic consistency assessment dimension, and the professional compliance assessment dimension.

5. The method according to claim 1, characterized in that, Also includes: Obtain multiple error annotation records obtained during historical quality inspection processes; At least one high-frequency error labeling type is determined based on the multiple error labeling records; High-risk special attention rules are constructed based on the aforementioned high-frequency error labeling types; Add the high-risk special attention rule to the preset referee task instruction; The high-frequency error labeling type is added to the multi-dimensional scoring rule.

6. The method according to claim 1, characterized in that, Based on the annotation quality score, all the pre-annotated data are divided into multiple datasets with different quality levels, including: All the pre-annotated data with quality scores greater than or equal to a preset score threshold are combined into a high-scoring dataset representing a relatively high quality level. All the pre-labeled data with quality scores lower than the preset score threshold are grouped into a low-score dataset representing a relatively low quality level.

7. The method according to claim 6, characterized in that, The sampling ratios corresponding to each quality level of the dataset are used to perform quality checks on each dataset, including: The high-scoring dataset is quality inspected based on the first sampling ratio; The low-scoring dataset is quality inspected based on a second sampling ratio; the first sampling ratio is less than the second sampling ratio.

8. The method according to claim 1, characterized in that, The step of inputting a preset annotation task instruction and multiple driving image data into a pre-annotation model to obtain pre-annotated data corresponding to each driving image data includes: For each driving image data, the driving image data and the preset annotation task instruction are input into the pre-annotation model to obtain the pre-annotation data corresponding to the driving image data; the pre-annotation data includes traffic light information, key target objects, detection boxes, and driving decisions.

9. A driving data annotation and optimization device, characterized in that, include: The pre-annotation module is used to input a preset annotation task instruction and multiple driving image data into the pre-annotation model to obtain pre-annotation data corresponding to each driving image data. The referee module is used to input the preset referee task instruction, the driving image data and its corresponding pre-annotated data into the pre-referee model for each driving image data, and obtain the annotation quality score corresponding to the pre-annotated data. The preset referee task instructions include multi-dimensional scoring rules and a set of veto conditions; The partitioning module is used to divide all the pre-labeled data into multiple datasets with different quality levels based on the labeled quality scores; The quality inspection module is used to perform quality inspection on each dataset based on the sampling ratio corresponding to each quality level.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the driving data annotation optimization method according to any one of claims 1-8.

Images