Data labeling method and device, vehicle, storage medium and program product

Abstract

The embodiment of the invention provides a data labeling method and device, a vehicle, a storage medium and a program product, and the method comprises the steps: obtaining modal data in at least one modal collected by a plurality of sensors disposed in the vehicle, and obtaining multi-modal data; the multi-modal data is input into an annotation model for annotation, at least one initial annotation result is obtained, the annotation model is obtained through training based on a multi-modal data sample collected by a sensor and an annotation result sample corresponding to the multi-modal data sample, and at least one annotation result is obtained; the initial labeling result is at least used for representing the position of at least one obstacle in the traffic area; in response to the fact that the confidence coefficient of the initial labeling result is lower than a confidence coefficient threshold value, obtaining an adjustment instruction; and in response to the adjustment instruction, adjusting the three-dimensional space label of the obstacle in the initial labeling result to obtain a target labeling result. The technical problem of low accuracy of data processing is solved.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and more specifically, to a data annotation method, apparatus, vehicle, storage medium, and program product. Background Technology

[0002] Currently, with the development of autonomous driving technology, vehicles require a large amount of high-quality multimodal data, such as images, videos, LiDAR point clouds, and semantic labels, to support the training of perception, decision-making, and control models.

[0003] In related technologies, data production and annotation are typically accomplished through manual annotation and partially rule-driven automated annotation tools. Manual methods offer advantages such as high operability and flexibility, but they suffer from low annotation efficiency and high costs when dealing with large-scale autonomous driving datasets. Some automated annotation tools utilize rules or traditional machine learning methods to achieve semi-automated annotation, but their ability to handle complex traffic scenarios and long-tail data is limited, still requiring significant manual intervention. Therefore, this method suffers from the technical problem of low data processing accuracy.

[0004] There is currently no good solution to the above problems. Summary of the Invention

[0005] This application provides a data annotation method, apparatus, vehicle, storage medium, and program product to at least solve the technical problem of low accuracy in data processing.

[0006] According to one aspect of the embodiments of this application, a data annotation method is provided. The method may include: acquiring modal data in at least one modality collected by multiple sensors deployed in a vehicle to obtain multimodal data, wherein the multimodal data is used to characterize the traffic state when the vehicle is located in a traffic area, and the traffic area is within a target area; inputting the multimodal data into an annotation model for annotation to obtain at least one initial annotation result, wherein the annotation model is trained based on multimodal data samples collected by sensors and annotation result samples corresponding to the multimodal data samples, and the initial annotation result is used to characterize the location of at least one obstacle in the traffic area; in response to the confidence level of the initial annotation result being lower than a confidence level threshold, obtaining an adjustment instruction; in response to the adjustment instruction, adjusting the three-dimensional spatial labels of the obstacles in the initial annotation result to obtain a target annotation result, wherein the three-dimensional spatial labels are used to characterize the location and size of the obstacles.

[0007] Furthermore, the multimodal data is input into the annotation model for annotation to obtain at least one initial annotation result, including: performing time-series alignment of the multimodal data according to timestamps; correcting the time-series aligned multimodal data to obtain multiple corrected data; and inputting the multiple corrected data into the annotation model for annotation to obtain at least one initial annotation result.

[0008] Furthermore, multiple correction data are input into the annotation model for annotation to obtain at least one initial annotation result, including: inputting prompt information and multiple correction data into the annotation model to obtain at least one initial annotation result, wherein the prompt information is used to prompt the annotation model for obstacles to be annotated.

[0009] Furthermore, in response to the confidence level of the initial annotation result being lower than the confidence level threshold, an adjustment instruction is obtained, including: in response to the confidence level of the initial annotation result being lower than the confidence level threshold, annotating the initial annotation result in at least one modal data corresponding to the initial annotation result; displaying the annotated modal data in the display interface; obtaining the operation data in the display interface; and converting the operation data into an adjustment instruction.

[0010] Furthermore, the method may also include: determining the similarity between multiple target annotation results corresponding to the obstacle at multiple consecutive time points, wherein the target annotation results correspond one-to-one with the time points; and in response to the similarity being lower than the similarity threshold, marking the obstacle and re-annotating the obstacle.

[0011] Furthermore, the method may further include: in response to the confidence level of the initial annotation result being lower than a confidence threshold, storing the multimodal data as multimodal data samples and the initial annotation result as annotation result samples in a database; in response to the number of multimodal data samples in the database exceeding a data volume threshold, selecting a target number of multimodal data samples and corresponding annotation result samples from the database; and updating the annotation model using the selected multimodal data samples and corresponding annotation result samples.

[0012] According to another aspect of the embodiments of this application, a data annotation apparatus is also provided, which may include: a first acquisition unit, configured to acquire modal data in at least one mode collected by multiple sensors deployed in a vehicle to obtain multimodal data, wherein the multimodal data is used to characterize the traffic state when the vehicle is located in a traffic area, and the traffic area is within a target area; an annotation unit, configured to input the multimodal data into an annotation model for annotation to obtain at least one initial annotation result, wherein the annotation model is trained based on multimodal data samples collected by sensors and annotation result samples corresponding to the multimodal data samples, and the initial annotation result is used to characterize the location of at least one obstacle in the traffic area; a second acquisition unit, configured to acquire an adjustment instruction in response to the confidence level of the initial annotation result being lower than a confidence level threshold; and an adjustment unit, configured to adjust the three-dimensional spatial labels of the obstacles in the initial annotation result in response to the adjustment instruction to obtain a target annotation result, wherein the three-dimensional spatial labels are used to characterize the location and size of the obstacles.

[0013] According to another aspect of the embodiments of this application, a vehicle is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods of various embodiments of this application when it runs.

[0014] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to run the methods of various embodiments of this application.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that implements the methods of various embodiments of this application when run by a processor.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which is executed by a processor to implement the methods in various embodiments of this application.

[0017] According to another aspect of the embodiments of this application, a computer program is also provided, which is executed by a processor to implement the methods of the various embodiments of this application.

[0018] In this embodiment, modal data in at least one mode is acquired from multiple sensors deployed in the vehicle to obtain multimodal data. This multimodal data characterizes the traffic state when the vehicle is located within a traffic area, which is within a target area. The multimodal data is input into a labeling model for annotation, resulting in at least one initial labeling result. The labeling model is trained based on multimodal data samples acquired by the sensors and corresponding labeling result samples. The initial labeling result characterizes the location of at least one obstacle in the traffic area. In response to the initial labeling result having a confidence level lower than a confidence threshold, an adjustment command is obtained. In response to the adjustment command, the three-dimensional spatial labels of the obstacles in the initial labeling result are adjusted to obtain a target labeling result. The three-dimensional spatial labels characterize the location and size of the obstacles. In other words, in this application, multimodal data is acquired, and the multimodal data is labeled using a labeling model to obtain initial labeling results. If the confidence level of the initial labeling results is lower than the confidence level threshold, the initial labeling results that are lower than the confidence level threshold can be directly adjusted, thereby achieving the technical effect of improving the accuracy of data processing and solving the technical problem of low accuracy in data processing. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart of a data annotation method according to an embodiment of this application;

[0021] Figure 2 This is a flowchart of an intelligent annotation and quality control method for autonomous driving data based on a large model, according to an embodiment of this application.

[0022] Figure 3 This is a schematic diagram of a data annotation device according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] According to an embodiment of this application, a method embodiment for data annotation is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0026] Currently, with the development of autonomous driving technology, vehicles require a large amount of high-quality multimodal data, such as images, videos, LiDAR point clouds, and semantic labels, to support the training of perception, decision-making, and control models.

[0027] In related technologies, data production and annotation are typically accomplished through manual annotation and partially rule-driven automated annotation tools. Manual methods offer advantages such as high operability and flexibility, but they suffer from low annotation efficiency and high costs when dealing with large-scale autonomous driving datasets. Some automated annotation tools utilize rules or traditional machine learning methods to achieve semi-automated annotation, but their ability to handle complex traffic scenarios and long-tail data is limited, still requiring significant manual intervention.

[0028] As can be seen from the above, manual annotation requires a large amount of manpower, especially in 3D point clouds and complex dynamic traffic scenarios, resulting in long annotation cycles and high costs. Different annotators have subjective differences, leading to a lack of consistency in annotation results and affecting the accuracy of model training. Traditional data collection and annotation methods cannot guarantee coverage of long-tail scenarios such as extreme weather and abnormal traffic events. Existing annotation methods lack intelligent quality detection and feedback mechanisms, resulting in the widespread existence of noisy labels, missing labels, or incorrect labels, which in turn affects the convergence and generalization ability of autonomous driving models. Therefore, the above methods have the technical problem of low data processing accuracy.

[0029] In this embodiment, multimodal data is acquired, and the multimodal data is labeled using a labeling model to obtain initial labeling results. If the confidence level of the initial labeling results is lower than the confidence level threshold, the initial labeling results that are lower than the confidence level threshold can be directly adjusted, thereby achieving the technical effect of improving the accuracy of data processing and solving the technical problem of low accuracy in data processing.

[0030] This embodiment provides a data annotation method. Figure 1 This is a flowchart of a data annotation method according to an embodiment of this application. For example... Figure 1 As shown, the method may include the following steps.

[0031] Step S102: Obtain modal data in at least one mode from multiple sensors deployed in the vehicle to obtain multimodal data. The multimodal data is used to characterize the traffic state when the vehicle is located in a traffic area, and the traffic area is within the target area.

[0032] In the technical solution provided in step S102 of this application, the aforementioned sensor can be a multimodal sensor, including but not limited to devices such as high-definition cameras, surround-view cameras, millimeter-wave radar, and lidar, which can be used to collect different modal data to obtain multimodal data. The aforementioned multimodal data can be used to provide content such as images, videos, point clouds, velocity, acceleration, and location information, and may include but not limited to: images, videos, lidar point clouds, semantic tags, and other data.

[0033] Optionally, a range of high-performance sensors can be installed on autonomous vehicles, including but not limited to high-definition cameras, surround-view cameras, millimeter-wave radar, lidar (LiDAR), inertial measurement units (IMUs), and global positioning systems (GPS). These sensors can be used to perceive different types of information, such as images, videos, point cloud data, speed, acceleration, and geographic location.

[0034] Optionally, the aforementioned modal data can refer to data derived from different sensing principles, such as visual modalities (images, videos), radar modalities (millimeter waves, infrared), and laser modalities (LiDAR point clouds). It should be noted that this step is applicable even in the case of a single modality (e.g., using only visual data). However, in this embodiment, to obtain richer and more accurate environmental information, data from multiple modalities can be collected simultaneously to obtain multimodal data. The complementarity of multimodal data can be utilized to enhance the overall perception capability.

[0035] Optionally, by integrating an onboard sensor network, multimodal data can be efficiently collected and integrated, providing a high-quality input source for subsequent intelligent annotation using large models.

[0036] Optionally, the aforementioned traffic area can be urban roads, highways, rural paths, tunnels, bridges, intersections, etc., and can be an area where vehicles are currently traveling. The aforementioned target area can be an area within a certain range in front of the vehicle, or an area that affects the vehicle's driving status.

[0037] Step S104: Input the multimodal data into the annotation model for annotation to obtain at least one initial annotation result. The annotation model is trained based on the multimodal data samples collected by the sensor and the annotation result samples corresponding to the multimodal data samples. The initial annotation result is used to characterize the location of at least one obstacle in the traffic area.

[0038] In the technical solution provided in step S104 of this application, the annotation model can be a multimodal large model, which can be trained based on multimodal data samples collected by sensors and the corresponding annotation result samples. It can be used to identify multimodal data to output initial annotation results. The initial annotation results can be used to annotate objects in traffic areas and the confidence level of those objects. They can be used to determine the location, shape, and other information of the labeled objects, and can include, but are not limited to, detection boxes, categories, and spatial location information of objects such as pedestrians, bicycles, cars, motorcycles, traffic lights, and traffic signs on roads. The objects can include static and dynamic objects, and can include, but are not limited to, pedestrians, bicycles, cars, motorcycles, traffic lights, and traffic signs on roads.

[0039] Optionally, after acquiring multimodal data, a multimodal large model can be used, which can be fine-tuned based on a pre-trained model. The input of the pre-trained model is standardized multimodal data, such as images and point cloud sequences, and the output can include object detection boxes, semantic segmentation masks, instance segmentation results, traffic element recognition (such as traffic lights, signs, and road sign information), and scene-level text descriptions.

[0040] Optionally, the aforementioned annotation model can be trained using deep learning techniques based on a large number of multimodal data samples and corresponding annotation result samples. At this stage, the annotation model learns to map various complex scenarios to specific tasks such as object detection, semantic segmentation, and instance segmentation, which are the core requirements for autonomous driving data annotation. The training samples can include various traffic scenarios, such as urban blocks, traffic flow on highways, complex intersections, and adverse weather conditions, as well as annotation information for various objects in these scenarios, such as the location, category, and size of vehicles, pedestrians, traffic signs, and road markings.

[0041] Optionally, the annotation model fully leverages the complementary characteristics of multimodal data such as images, videos, and point clouds. For example, images provide color and texture information, while point clouds provide depth and 3D spatial information. The fusion of these multimodal data makes the model more accurate and robust in identifying and locating obstacles. The annotation model can process and integrate data from different sensors simultaneously, thereby obtaining more comprehensive and accurate obstacle location information.

[0042] Optionally, when multimodal data is input into the annotation model, the model can process the multimodal data to output annotation information, i.e., "initial annotation results." These initial annotation results can be directly used to characterize the location of obstacles in a traffic area, as well as the status of other potentially important traffic elements, such as lane line positions and traffic light status. Furthermore, the initial annotation results are not limited to obstacle localization; they can also include obstacle classification (e.g., pedestrians, vehicles), semantic segmentation (distinguishing between road and non-road areas), instance segmentation (identifying different individuals within the same category), and scene-level descriptions (e.g., sunny days, rainy days, traffic congestion, etc.).

[0043] Alternatively, since the annotation model is trained on a wide variety of datasets, it can generalize using the learned patterns and concepts in new scenarios, and can provide reasonable annotation results even for obstacle types or scene features that have not appeared precisely in the training set.

[0044] Optionally, the initial annotation results described above can provide crucial data support for training autonomous driving models. Automatic annotation significantly reduces manual workload and accelerates data processing, enabling autonomous driving systems to absorb new knowledge more quickly and improve driving safety in various complex traffic environments.

[0045] Through the above step S104, the intelligent annotation system based on the large model can efficiently process multimodal data and generate detailed initial annotation results. These results cover the accurate identification and location annotation of obstacles and other important elements in the traffic area, laying a solid foundation for subsequent human-machine collaborative correction and quality control.

[0046] Step S106: In response to the initial annotation result having a confidence level lower than the confidence level threshold, an adjustment instruction is obtained.

[0047] In the technical solution provided in step S106 of this application, a confidence score can be assigned to each annotation result. When the confidence score of the annotation result is higher than the threshold, it can be directly adopted. When the confidence score is in the fuzzy range, the task can be assigned to a human reviewer. The reviewer only needs to make corrections based on the existing large model output by inputting adjustment instructions. These adjustment instructions can be input by the operator through a display interface or other terminal, and can be used to adjust the position, shape, and other information of the initial annotation result.

[0048] Optionally, the aforementioned confidence level can be used to determine the degree of certainty in the initial annotation results. The annotation model assigns a confidence score to each initial annotation result output from the multimodal data, which can be used to reflect the model's estimation of the correctness of the annotation results. For example, if the model is very certain that the image contains a car, then the corresponding car bounding box will have a high confidence score.

[0049] Optionally, a confidence threshold can be preset according to different application scenarios and quality requirements. This confidence threshold can be used to determine whether the annotation results are reliable enough. Initial annotation results with a confidence level lower than this threshold can be considered uncertain or potentially erroneous, requiring further review or correction. In such cases, adjustment instructions for the initial annotation results that need correction can be obtained, and the correction process for the initial annotation results can be completed according to these instructions.

[0050] Optionally, when the confidence level of the initial annotation results is lower than a set confidence threshold, the system can automatically identify and filter these results, marking them as samples awaiting review or correction. These samples will be further processed instead of being directly used for model training. This step effectively reduces the total amount of manual correction, focusing only on those parts that most require intervention, thereby improving data processing efficiency.

[0051] Optionally, for initial annotation results with low confidence, adjustment instructions can be triggered. These instructions may include, but are not limited to, task requests received from human reviewers, model inference instructions, or data augmentation instructions. For example, when the annotation model is uncertain about the location of an obstacle, an instruction can be generated instructing a human annotator to check and correct the obstacle's position. After receiving the instruction, the human reviewer can directly drag and drop on the image or point cloud using a visual annotation tool to input the adjustment instructions. Following these instructions, the annotation results can be updated in real time, and the corresponding 3D spatial labels can be adjusted synchronously to ensure the accuracy of the annotation.

[0052] In this embodiment, a confidence assessment and adjustment instruction mechanism enables intelligent review and correction of the initial annotation results, ensuring that the final output annotation data meets high-quality standards. This process not only enhances the authenticity of the data but also promotes the self-improvement of the model.

[0053] Step S108: In response to the adjustment instruction, the three-dimensional spatial labels of obstacles in the initial annotation result are adjusted to obtain the target annotation result, wherein the three-dimensional spatial labels are used to characterize the location and size of the obstacles.

[0054] In the technical solution provided by step S108 of this application, the three-dimensional spatial label can be used to characterize the spatial position and size of the obstacle, and can be used to accurately locate the obstacle in the three-dimensional coordinate system and determine the size and shape of the obstacle.

[0055] Optionally, by adjusting the three-dimensional spatial labels, it can be ensured that the obstacle information output by the labeled model is accurate and meets the high-precision requirements of the autonomous driving system for environmental perception.

[0056] Optionally, when the confidence level of the initial annotation result is detected to be lower than a predetermined confidence threshold in step S106, it can be determined that the annotation model is not entirely certain about the location, shape, or size of certain obstacles. Low confidence may be caused by various factors, such as obstacles being in unfavorable lighting conditions, severe weather, or appearing in complex traffic scenarios that the annotation model has not yet fully grasped. In this case, an adjustment instruction can be generated, which may include instructions for manual review or model re-inference of the annotation results with low confidence.

[0057] Optionally, after the adjustment instruction is received, the corresponding multimodal data can be presented to a human reviewer. The reviewer can directly drag and drop on the corresponding modal data, such as images or point cloud data, using visual annotation tools to adjust the 3D spatial information of obstacles, including their bounding boxes, dimensions, and positions. This interactive correction method allows the reviewer to manually adjust the 3D spatial labels based on actual observations until satisfactory accuracy is achieved.

[0058] Optionally, if a human reviewer believes that the low confidence level of the annotation model is due to the model not having sufficiently learned a certain scene or obstacle type, the reviewer can choose to send the multimodal data back to the annotation model for further reasoning. In this case, the annotation model can re-annotate the multimodal data.

[0059] Optionally, the above target annotation results can be the final version after correction and optimization of the initial annotation results. During the adjustment of 3D spatial labels, the reviewer or annotation model will mark the position, size, and shape of obstacles as accurately as possible to ensure the accuracy of each label.

[0060] Optionally, when adjusting the 3D spatial labels of obstacles in the initial annotation results, the consistency and robustness of the entire dataset can also be considered. That is, when adjusting the 3D spatial labels, not only should the accuracy of individual samples be considered, but also the adjusted labels should be kept consistent with the annotations of neighboring samples and obstacles of the same type, avoiding unnecessary differences in annotation style or scale. At the same time, through repeated corrections and model optimization, the stability of the annotation results can be improved when facing unknown or complex scenes.

[0061] In this embodiment, step S108, in response to adjustment instructions, specifically performs detailed corrections on the initial annotation results with low confidence, particularly the 3D spatial labels of obstacles. This step fully demonstrates the advantages of human-machine collaboration, utilizing both the automation and computational capabilities of the model and the professional judgment and correction capabilities of human review. The final target annotation results have higher accuracy and consistency, providing strong data support for the continuous advancement of autonomous driving technology. This process not only improves data quality but also indirectly promotes the iterative optimization of large models, forming a virtuous cycle of data production and annotation.

[0062] Optionally, the final target annotation results can be used to identify the movement of obstacles in the vehicle and the relationship between the obstacles and the vehicle, thereby enabling vehicle path planning.

[0063] Through steps S102 to S108, modal data in at least one mode is acquired from multiple sensors deployed in the vehicle to obtain multimodal data. This multimodal data characterizes the traffic state when the vehicle is located within a traffic area, which is within a target area. The multimodal data is then input into a labeling model for annotation, yielding at least one initial labeling result. This labeling model is trained based on multimodal data samples acquired by the sensors and corresponding labeling result samples. The initial labeling result characterizes the location of at least one obstacle in the traffic area. In response to the initial labeling result having a confidence level lower than a confidence threshold, an adjustment command is obtained. In response to the adjustment command, the three-dimensional spatial labels of the obstacles in the initial labeling result are adjusted to obtain the target labeling result. The three-dimensional spatial labels characterize the location and size of the obstacles. In other words, in this application, multimodal data is acquired, and the multimodal data is labeled using a labeling model to obtain initial labeling results. If the confidence level of the initial labeling results is lower than the confidence level threshold, the initial labeling results that are lower than the confidence level threshold can be directly adjusted, thereby achieving the technical effect of improving the accuracy of data processing and solving the technical problem of low accuracy in data processing.

[0064] The above-mentioned method of this application will be further described below.

[0065] As an optional implementation, step S104 involves inputting multimodal data into a labeling model for labeling to obtain at least one initial labeling result, including: performing time-series alignment of the multimodal data according to timestamps; correcting the time-series aligned multimodal data to obtain multiple corrected data; and inputting the multiple corrected data into a labeling model for labeling to obtain at least one initial labeling result.

[0066] In this embodiment, multiple data acquisition methods are supported. For example, the vehicle-mounted acquisition terminal may include high-definition cameras, surround-view cameras, millimeter-wave radar, lidar, IMU, and GPS positioning devices to obtain multimodal data. The multimodal data acquired by this multimodal sensor can be used to provide image, video, point cloud, velocity, acceleration, and location information. To ensure the effectiveness of subsequent annotation and training, after acquiring the multimodal data, spatiotemporal synchronization, distortion correction, noise filtering, and formatting can be performed on the acquired multimodal data to uniformly generate standard input samples adapted to large model inference.

[0067] Optionally, multimodal data can be time-aligned according to timestamps. This ensures a one-to-one correspondence between image frames and point cloud frames through time alignment algorithms, avoiding data mismatch caused by different sensor sampling frequencies. At the same time, metadata can be added to the collected samples (i.e., multimodal data), such as collection time, location, weather conditions, and road scene type. This metadata can provide additional conditions for the active learning of subsequent long-tail samples.

[0068] Optionally, modal data from different sensors can be fused to generate multimodal data. A temporal alignment algorithm can be used to correspond image frames with point cloud frames, eliminating data mismatch caused by different sampling frequencies between sensors. Distortion correction can be applied to the image to remove image distortion caused by wide-angle lenses, improving the accuracy of target detection. Noise filtering can be performed on the point cloud data to remove invalid or distant points, improving obstacle localization accuracy. This yields corrected data after alignment and correction processing, which can then be transmitted to a labeling model for annotation to obtain at least one initial annotation result.

[0069] Optionally, the preprocessed multimodal data (i.e., corrected data) can be used to characterize the traffic conditions in the traffic area where the vehicle is currently traveling, including dynamic and static elements such as other vehicles, pedestrians, bicycles, static obstacles, lane lines, traffic lights, and traffic signs on the road.

[0070] Optionally, multimodal data in autonomous driving systems, such as images, videos, and point cloud data, can be collected by multiple different types of sensors. Since the sampling frequencies and delays of these sensors may differ, directly using this data can lead to temporal mismatches, affecting the accuracy of data annotation. Therefore, the collected multimodal data can be time-aligned based on the sensor timestamps. A time alignment algorithm can be used, that is, based on the sensor timestamp information, to adjust the data from different modalities to the same time reference, ensuring a one-to-one correspondence between image frames and point cloud frames, and eliminating data mismatch problems caused by differences in the sampling frequencies of different sensors.

[0071] Optionally, after aligning the multimodal data, the aligned multimodal data can be corrected. The correction process may include, but is not limited to, distortion correction, noise filtering, and data standardization. Distortion correction can be performed on image data to eliminate image distortion caused by wide-angle lenses or other optical effects, making target detection more accurate. For point cloud data, noise filtering is required to remove invalid or distant points, improving obstacle localization accuracy. After the above processing, the data can be standardized to convert it into a model-recognizable format, such as adjusting image size, normalizing pixel values, and organizing point cloud coordinates, so that it can be smoothly input into the labeled model for processing.

[0072] Optionally, the annotation model is a deep learning model trained with a large number of multimodal data samples and their corresponding annotation result samples. It can process multiple modalities of data such as images, videos, and point clouds, and output various annotation information including object detection boxes, semantic segmentation, instance segmentation, and scene descriptions. The corrected data obtained after correction can be uniformly input into the annotation model. After inference, the model outputs at least one initial annotation result. This initial annotation result can be a preliminary judgment of the location of obstacles in the traffic area based on the input data, including but not limited to the location, category, and size information of vehicles, pedestrians, bicycles, and static obstacles.

[0073] Optionally, the annotation model can utilize the massive amount of general knowledge learned in the pre-training stage to make relatively accurate and reliable annotations in complex traffic scenarios. Especially when dealing with long-tail scenarios and rare events, the large model shows better generalization ability through pattern recognition and feature extraction compared to traditional rule-driven methods.

[0074] In this embodiment, temporal alignment, correction, and standardization of multimodal data create favorable conditions for subsequent intelligent annotation. The annotation model uses the optimized data to generate initial annotation results, providing a foundation for subsequent quality control and human-machine collaborative correction. The entire process demonstrates the close integration of data preparation and model application, and is a key link in building a high-efficiency, high-precision autonomous driving data production process.

[0075] As an optional implementation, multiple correction data are input into the annotation model for annotation to obtain at least one initial annotation result, including: inputting prompt information and multiple correction data into the annotation model to obtain at least one initial annotation result, wherein the prompt information is used to prompt the annotation model for obstacles to be annotated.

[0076] In this embodiment, the aforementioned prompt information can be pre-inputted information, which can be text or voice information, or a prompt message. It can be used to indicate obstacles to be labeled to the labeling model. For example, it could be "Please identify all traffic elements that affect driving safety and output the bounding box and category." It should be noted that this is only an example and there are no specific limitations on the type and content of the prompt information.

[0077] Optionally, the annotation model can achieve higher annotation accuracy through prompting engineering mechanisms. For example, a prompt such as "Please identify all traffic elements affecting driving safety and output bounding boxes and categories" can be added to the input information. Inputting the prompt information and preprocessed correction data into the annotation model allows it to focus on key elements (i.e., obstacles) such as lane lines, pedestrians, vehicles, cones, and traffic lights. Unlike traditional algorithms, this embodiment utilizes the knowledge transfer capabilities of a large model, enabling it to quickly adapt to differences in traffic scenarios across different countries and regions. For instance, traffic signs in Europe and America differ from those in Asia, and the large model can generalize using prior knowledge from a large-scale pre-training corpus.

[0078] Optionally, the aforementioned prompts provide contextual guidance for the annotation model, clarifying the specific types of obstacles or traffic elements that need to be focused on and annotated. For example, the prompt could be a brief description, such as "Please identify and annotate all vehicles and pedestrians." These prompts enable the model to focus on the most critical targets when processing data, avoiding the omission or mislabeling of other irrelevant elements.

[0079] In this embodiment, by providing additional semantic information to the annotation model, the prompts can help the model more accurately understand and annotate complex scenes. Especially when dealing with blurred or heavily occluded obstacles, the prompts can guide the model to fully utilize pre-trained contextual knowledge to make more reasonable judgments, thereby improving overall annotation accuracy. These prompts can be information generated from multimodal data or information pre-input into the annotation model.

[0080] Optionally, unlike traditional unguided annotation, intelligent annotation of large models with prompts can better adapt to the diversity and differences in cross-regional and cross-cultural environments. For example, with specific prompts, the model can learn to recognize traffic signs and road rules in different countries / regions, improving its robustness and generalization ability in global application.

[0081] Optionally, the corrected multimodal data includes diverse information such as images, videos, and point clouds. This data captures the state of the traffic environment from different angles and modalities, providing the model with more comprehensive input information. For example, image data can provide rich texture and color information, while point cloud data can reveal three-dimensional spatial structure. Combining the two helps the model to more accurately locate and identify obstacles. Different modalities of data have their own advantages and disadvantages in capturing environmental information. For example, image data performs excellently under good lighting conditions, but may be limited at night or in inclement weather; conversely, while point cloud data is not affected by lighting conditions, its ability to capture material and color information is relatively weak. Integrating this complementary information into the model allows the model to combine the advantages of various data types, compensate for the limitations of single-modal data, and thus improve the stability and accuracy of obstacle annotation.

[0082] In this embodiment, after receiving multimodal correction data and prompts, the annotation model can perform in-depth analysis and reasoning on the data based on its own knowledge and training experience to identify and annotate obstacles to be labeled. The output of the annotation model can include information such as the location, type, and size of the obstacles, which can be presented in the form of initial annotation results. Each obstacle is assigned a three-dimensional spatial label to accurately describe its specific location and shape in the traffic environment.

[0083] In this embodiment, the efficiency and accuracy of autonomous driving data annotation are improved by combining prompts with multimodal correction data input into the annotation model. The prompts provide additional semantic guidance, helping the model focus on key obstacles, while the fusion of multimodal data makes the annotation results more comprehensive and reliable. This strategy is particularly important in handling complex traffic scenarios, not only improving annotation speed but also ensuring the quality of the annotation results, providing a solid foundation for subsequent autonomous driving model training.

[0084] As an optional implementation, step S106, in response to the confidence level of the initial annotation result being lower than the confidence level threshold, obtains an adjustment instruction, including: in response to the confidence level of the initial annotation result being lower than the confidence level threshold, annotating the initial annotation result in at least one modal data corresponding to the initial annotation result; displaying the annotated modal data in the display interface; obtaining operation data in the display interface; and converting the operation data into an adjustment instruction.

[0085] In this embodiment, if the confidence level of the initial annotation result is lower than the confidence level threshold, the initial annotation result can be annotated in at least one modal data corresponding to the initial annotation result. The annotated modal data can be displayed on a display interface. This display interface can be a terminal display interface, such as the display interface of a computer, mobile phone, or tablet computer. Operation data in the display interface can be acquired, which can be drag-and-drop, deletion, or other operation data. The operation data can be converted into adjustment commands.

[0086] Optionally, this embodiment fully leverages the advantages of human-machine collaboration. A confidence score can be assigned to each initial annotation result. When the confidence score of the initial annotation result is higher than the confidence threshold, it can be directly adopted. When the confidence score is in the fuzzy range, i.e., lower than the confidence threshold, the initial annotation result can be assigned to a human reviewer. The reviewer only needs to make corrections based on the existing large model output, avoiding redundant operations of annotation from scratch. This mechanism greatly reduces the workload of manual intervention, transforming manual annotation from a full-scale operation to focused review. Furthermore, a visual annotation tool is provided, allowing reviewers to directly drag and modify on the image or point cloud to obtain operational data. By converting the operational data, adjustment instructions are obtained. Following these instructions, the initial annotation results are adjusted to obtain the target annotation results, thereby achieving the goal of real-time updating of annotation results and synchronous adjustment of the corresponding 3D spatial labels.

[0087] Optionally, when the annotation model outputs initial annotation results based on multimodal data, each annotation is accompanied by a confidence value, which represents the model's confidence in the correctness of the annotation. A confidence threshold can be set; any annotation below this threshold is considered to have low confidence and may require further verification or correction. For annotation results with low confidence, instead of simply discarding the initial annotation result, it is annotated in the corresponding modal data, that is, the model's output annotation is directly displayed on image, video, or point cloud data. This method provides human reviewers with an intuitive reference, allowing them to directly see the model's performance on specific data; at the same time, it clearly identifies which parts of the model need improvement or rethinking.

[0088] Optionally, the modal data marked for review can then be presented to human reviewers in a dedicated display interface. This interface is typically designed to be user-friendly, facilitating interactive corrections by reviewers. For example, reviewers can see images or point clouds with bounding boxes or segmentation labels, and adjust the position, size, and shape of these labels with simple mouse operations. This mechanism of direct operation on the display interface greatly simplifies the correction process and avoids cumbersome data import and export steps.

[0089] Optionally, when the reviewer modifies the annotations on the display interface, each operation is recorded, forming operation data. This data may include, but is not limited to, mouse movement trajectory, click position, drag direction and distance, etc. This operation data can be captured in real time and converted into adjustment instructions, which can be specific parameter adjustments, operation commands, or feedback signals to guide model learning. In this way, not only can the annotation of individual samples be corrected in real time, but the lessons learned during the correction process can also be fed back to the annotation model, enabling the model to make more accurate predictions when encountering similar samples in the future.

[0090] Optionally, the operational data accumulated during the correction process is not only a means of immediate adjustment, but also a driving force for the learning and improvement of large models. This data can be considered as challenging examples and incorporated into the model's active learning process. By continuously feeding corrected samples back to the model for retraining or fine-tuning, the model can gradually improve its generalization ability in complex scenarios and low-confidence situations, ultimately achieving continuous optimization of model performance. This is also one of the key steps in forming a self-evolving data loop, ensuring a dual improvement in data quality and model capabilities.

[0091] In this embodiment, through human-computer interaction, not only can the annotation results be corrected in real time, but the model can also be improved by self-improvement. This provides a more accurate, consistent and comprehensive dataset for training autonomous driving models, thereby promoting the intelligence and efficiency of the entire data production process.

[0092] As an optional implementation, the method may further include: determining the similarity between multiple target annotation results corresponding to obstacles at multiple consecutive time points, wherein the target annotation results correspond one-to-one with the time points; and, in response to a similarity lower than a similarity threshold, marking the obstacle and re-annotating the obstacle.

[0093] In this embodiment, at multiple consecutive time points, the similarity between multiple target annotation results corresponding to the obstacle can be determined. If the similarity is lower than the similarity threshold, it can be determined that the category or position of the obstacle in the subsequent frame has changed abruptly. The obstacle can then be marked and anomaly detection can be triggered, thereby re-annotating the obstacle.

[0094] Optionally, this embodiment also introduces a multi-dimensional quality detection mechanism. That is, the consistency of the trajectory of the same obstacle in multiple frames of images can be checked. If the category or position of the target changes abruptly between frames, anomaly detection and reprocessing can be triggered. In addition, a cross-validation model can be used to compare the annotation results of the large model (i.e., the target annotation results at consecutive time points). When the outputs of two independent models on the same sample differ significantly, the sample is marked as a high-risk sample and enters manual review. Furthermore, an adversarial example detection mechanism can be used, that is, by constructing perturbation samples to detect the stability of the large model in extreme scenarios. If the model is found to be insufficiently robust to certain samples, the model optimization process can be automatically triggered.

[0095] Optionally, autonomous driving data is typically collected in continuous time series, meaning that for the same obstacle, there will be multiple target annotation results at different time points. These results should change continuously and naturally over time, reflecting the dynamic motion state of the obstacle or changes in the environment. At multiple consecutive moments, the same obstacle can be tracked to ensure that each annotation targets the same entity. This can be achieved through association algorithms; for example, the appearance features, location information, or movement trends of obstacles can be used to determine whether multiple obstacles belong to the same object. Furthermore, a suitable similarity metric can be defined to evaluate the differences in target annotation results at different time points. The comparison process can involve comparing the consistency of obstacle position, size, and category. For example, the Intersection over Union (IoU) ratio of bounding boxes can be calculated, or the distance and size changes of the obstacle's center point at two time points can be compared. For each obstacle, the similarity between the corresponding target annotation results can be calculated at multiple consecutive moments. If the similarity is lower than a pre-set similarity threshold, it indicates that the obstacle's annotation has undergone unreasonable abrupt changes or fluctuations in a short period, potentially indicating annotation errors or model instability.

[0096] Optionally, when an obstacle is found to have excessively low similarity between its target annotation results at adjacent time points, the obstacle can be automatically marked as a potentially inconsistent annotation sample. For the marked obstacle, a re-annotation process can be triggered. This re-annotation process can be implemented in two ways: first, by sending the obstacle and related multimodal data back to the annotation model for complex inference; second, by submitting the sample for manual review and correction by professional annotators. Complex inference helps the model re-evaluate under uncertain circumstances, while manual correction is used to correct complex or rare scenarios that the model may not be able to handle.

[0097] Optionally, during the re-labeling process, samples with similarity below a threshold are considered "hard examples" and collected into the system's hard example pool. The hard example pool is an important component of the active learning strategy. Samples can be periodically extracted from the pool and used as part of the training set to retrain or fine-tune the labeled model, thereby gradually improving the model's performance when handling such complex or marginal cases.

[0098] In this embodiment, by comparing the similarity of obstacle annotation results at multiple consecutive time points, potential errors in the annotation can be identified and corrected in a timely manner, ensuring the annotation quality and consistency of the dataset. This time-series-based consistency verification mechanism, combined with the dual protection of model complex inference and manual review, effectively improves the reliability of autonomous driving data production, which is of great significance for improving the performance and safety of autonomous driving systems in practical applications. Simultaneously, the combination of difficult example collection and active learning promotes the model's self-improvement, making it more robust and accurate in the face of complex and ever-changing traffic environments.

[0099] As an optional implementation, the method may further include: in response to the confidence level of the initial annotation result being lower than a confidence threshold, storing the multimodal data as multimodal data samples and the initial annotation result as annotation result samples in a database; in response to the number of multimodal data samples in the database exceeding a data volume threshold, selecting a target number of multimodal data samples and corresponding annotation result samples from the database; and updating the annotation model using the selected multimodal data samples and corresponding annotation result samples.

[0100] In this embodiment, if the confidence level of the initial annotation result is lower than the confidence level threshold, the multimodal data can be used as multimodal data samples, and the initial annotation result can be used as annotation result samples, and stored in the database as training samples. Through this method, multiple training samples can be collected. When the number of multimodal data samples in the database exceeds the data volume threshold, a target number of multimodal data samples and their corresponding annotation result samples can be selected from the database. The annotation model is then updated using the selected multimodal data samples and their corresponding annotation result samples. The aforementioned data volume threshold can be a pre-set data volume.

[0101] Optionally, this embodiment achieves continuous optimization of the large model through a hard example pool mechanism. Low-confidence samples, abnormal samples, and long-tail scene samples are uniformly stored in the hard example pool, and these samples are periodically extracted as part of the training set to retrain or fine-tune the large model. By continuously introducing new scenes and hard example data, the annotation capability of the large model will be continuously improved.

[0102] For example, when the accuracy of the labeled model decreases under adverse weather conditions such as rain or snow, relevant training data samples can be prioritized for the hard example pool to improve the model's generalization ability. This closed-loop mechanism ensures that the data production system has self-evolutionary characteristics and avoids performance degradation caused by environmental changes or data distribution drift.

[0103] In this embodiment, the system architecture may include a data acquisition module, a large model annotation module, a human collaboration module, a quality control module, and a feedback learning module. The data acquisition module can complete the acquisition and preprocessing of multimodal sensor data; the large model annotation module performs intelligent annotation based on pre-trained and fine-tuned models; the human collaboration module is used for manual correction of samples with insufficient confidence; the quality control module performs consistency verification, cross-validation, and anomaly detection on the labeled data; and the feedback learning module continuously optimizes the performance of the large model based on active learning and a hard example pool mechanism. The modules are interconnected through a data bus to form a complete closed-loop system for autonomous driving data production.

[0104] Optionally, the quality control and feedback closed-loop framework proposed in this embodiment can monitor the accuracy of the labeled data in real time, and ensure that the final output data meets the requirements of high precision and high consistency through automatic feedback and manual review mechanisms. The advantage of this framework is that it has a high degree of automation, while retaining a backup mechanism for manual review, thus achieving a balance between efficiency and reliability.

[0105] In this example, through an active learning and feedback closed-loop mechanism, it can continuously adapt to complex and ever-changing road scenarios, ultimately providing high-quality data support for the training of autonomous driving models and promoting the rapid development of intelligent driving technology.

[0106] In this embodiment, multimodal data is acquired, and the multimodal data is labeled using a labeling model to obtain initial labeling results. If the confidence level of the initial labeling results is lower than the confidence level threshold, the initial labeling results that are lower than the confidence level threshold can be directly adjusted, thereby achieving the technical effect of improving the accuracy of data processing and solving the technical problem of low accuracy in data processing.

[0107] The technical solutions of the embodiments of the present invention will be illustrated below with reference to preferred embodiments.

[0108] In this embodiment, a method for intelligent annotation and quality control of autonomous driving data based on a large model is proposed. This method preprocesses multimodal raw data from vehicle sensors to achieve spatiotemporal synchronization and standardization. Subsequently, a visual-language multimodal large model is used to perform preliminary automatic annotation of the data, generating object categories, bounding boxes, semantic segmentation, and scene descriptions. The system combines the large model annotation results with manual review to achieve efficient human-machine collaborative annotation. Intelligent quality control is performed through consistency verification, cross-model validation, and adversarial detection, and when problems are detected, the data is automatically fed back to model re-inference or manual review. Finally, combined with an active learning mechanism, the large model annotator is continuously iterated and optimized to form a self-evolving data closed loop, thereby achieving the technical effect of improving the accuracy of data processing and solving the technical problem of low data processing accuracy.

[0109] Compared with existing technologies, this invention can significantly improve the efficiency of autonomous driving data production and annotation, and greatly reduce manual workload; it improves the consistency of annotation results and reduces subjective differences by using a unified large model inference mechanism; it effectively ensures the accuracy and reliability of annotation data through intelligent quality detection and feedback mechanisms; and it relies on active learning strategies to ensure that long-tail scene samples are given priority for supplementation and optimization, thereby improving the generalization ability and robustness of autonomous driving models in complex scenarios.

[0110] Figure 2 This is a flowchart of a method for intelligent annotation and quality control of autonomous driving data based on a large model, according to an embodiment of this application. Figure 2 As shown, the method may include the following steps.

[0111] Step S202, Data Acquisition and Preprocessing.

[0112] In this embodiment, multiple data acquisition methods are supported. For example, the vehicle-mounted acquisition terminal may include high-definition cameras, surround-view cameras, millimeter-wave radar, lidar, IMU, and GPS positioning devices. These multimodal sensors can provide images, videos, point clouds, velocity, acceleration, and location information. To ensure the effectiveness of subsequent annotation and training, this invention performs spatiotemporal synchronization, distortion correction, noise filtering, and formatting on the raw data after acquisition, uniformly generating standard input samples adapted for large model inference.

[0113] Step S204: Label the collected data using a large model.

[0114] In this embodiment, a multimodal large model is used, which is fine-tuned based on a pre-trained model. The input of the multimodal large model can be a standardized image and point cloud sequence, and the output includes initial annotation results such as object detection boxes, semantic segmentation masks, instance segmentation results, traffic element recognition (such as traffic lights, signs, road signs information) and scene-level text descriptions.

[0115] Step S206, Human-machine collaborative correction step.

[0116] In this embodiment, the advantages of human-machine collaboration are fully utilized. A confidence score is assigned to each annotation result. When the confidence score of the annotation result is higher than the threshold, it can be directly adopted. When the confidence score is in the fuzzy range, the task will be assigned to a human reviewer. The reviewer only needs to make corrections based on the existing large model output.

[0117] Step S208, Intelligent Quality Control.

[0118] In this embodiment, a multi-dimensional quality detection mechanism is introduced. For example, the consistency of the trajectory of the same object in multiple frames of images can be checked. If the category or position of the target changes abruptly in the previous and next frames, anomaly detection and reflow processing can be triggered.

[0119] Step S210: Active learning and feedback closed loop.

[0120] In this embodiment, the large model is continuously optimized through the hard example pool mechanism. Low confidence samples, abnormal samples, and long-tail scene samples are uniformly stored in the hard example pool, and these samples are periodically extracted as part of the training set to retrain or fine-tune the large model. By continuously introducing new scenes and hard example data, the annotation capability of the large model will be continuously improved.

[0121] In this embodiment, by introducing large models for intelligent annotation and quality control, an efficient, accurate and scalable autonomous driving data production process is achieved, solving the problems of low efficiency, poor consistency, insufficient scene coverage and lack of quality in the existing autonomous driving data production and annotation process.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0123] According to the embodiments of this application, an embodiment of a data annotation system is also provided. It should be noted that the system can be used to run the above-mentioned data annotation method.

[0124] According to an embodiment of this application, a data annotation device embodiment is also provided. It should be noted that the device can be used to run the above-described data annotation method.

[0125] Figure 3 This is a schematic diagram of a data annotation device according to an embodiment of this application. Figure 3 As shown, the data annotation device may include: a first acquisition unit 302, an annotation unit 304, a second acquisition unit 306, and an adjustment unit 308.

[0126] The first acquisition unit 302 is used to acquire modal data in at least one mode collected by multiple sensors deployed in the vehicle to obtain multimodal data, wherein the multimodal data is used to characterize the traffic state when the vehicle is located in a traffic area, and the traffic area is within the target area.

[0127] The annotation unit 304 is used to input multimodal data into the annotation model for annotation to obtain at least one initial annotation result. The annotation model is trained based on multimodal data samples collected by sensors and annotation result samples corresponding to the multimodal data samples. The initial annotation result is used to characterize the location of at least one obstacle in the traffic area.

[0128] The second acquisition unit 306 is used to acquire an adjustment instruction in response to the confidence level of the initial annotation result being lower than the confidence level threshold.

[0129] The adjustment unit 308 is used to adjust the three-dimensional spatial labels of obstacles in the initial annotation result in response to the adjustment command, so as to obtain the target annotation result. The three-dimensional spatial labels are used to characterize the location and size of the obstacle.

[0130] Furthermore, the annotation unit 304 may include: an alignment module for performing temporal alignment of the multimodal data according to the timestamp; a correction module for correcting the temporally aligned multimodal data to obtain multiple corrected data; and a first annotation module for inputting the multiple corrected data into the annotation model for annotation to obtain at least one initial annotation result.

[0131] Furthermore, the first annotation module may include an input submodule, used to input prompt information and multiple correction data into the annotation model to obtain at least one initial annotation result, wherein the prompt information is used to indicate the obstacle to be annotated to the annotation model.

[0132] Furthermore, the second acquisition unit 306 includes: a second annotation module, used to annotate the initial annotation result in at least one modal data corresponding to the initial annotation result in response to the confidence level of the initial annotation result being lower than the confidence level threshold; a display module, used to display the annotated modal data in a display interface; and an acquisition module, used to acquire the operation data in the display interface and convert the operation data into adjustment instructions.

[0133] Furthermore, the device may also include: a determining unit, configured to determine the similarity between multiple target annotation results corresponding to an obstacle at multiple consecutive time points, wherein the target annotation results correspond one-to-one with the time points; and a marking unit, configured to mark the obstacle and re-annotate the obstacle in response to a similarity lower than a similarity threshold.

[0134] Furthermore, the device may further include: a processing unit, configured to, in response to the confidence level of the initial annotation result being lower than a confidence threshold, treat the multimodal data as multimodal data samples and the initial annotation result as annotation result samples, and store them in a database; a selection unit, configured to, in response to the number of multimodal data samples in the database exceeding a data volume threshold, select a target number of multimodal data samples and corresponding annotation result samples from the database; and an update unit, configured to update the annotation model using the selected multimodal data samples and corresponding annotation result samples.

[0135] In the data annotation device of this embodiment, a first acquisition unit acquires modal data in at least one mode collected by multiple sensors deployed in the vehicle to obtain multimodal data. The multimodal data is used to characterize the traffic state when the vehicle is located in a traffic area, and the traffic area is within the target area. A annotation unit inputs the multimodal data into an annotation model for annotation to obtain at least one initial annotation result. The annotation model is trained based on multimodal data samples collected by the sensors and annotation result samples corresponding to the multimodal data samples. The initial annotation result is used to characterize the location of at least one obstacle in the traffic area. A second acquisition unit acquires an adjustment command in response to the confidence level of the initial annotation result being lower than the confidence level threshold. An adjustment unit adjusts the three-dimensional spatial labels of the obstacles in the initial annotation result in response to the adjustment command to obtain the target annotation result. The three-dimensional spatial labels are used to characterize the location and size of the obstacles, thereby achieving the technical effect of improving the accuracy of data processing and solving the technical problem of low data processing accuracy.

[0136] Embodiments of this application also provide a vehicle, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods described in various embodiments of this application.

[0137] Embodiments of this application also provide a computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0138] Embodiments of this application also provide a computer program product, including a computer program that implements the methods of various embodiments of this application when run by a processor.

[0139] Embodiments of this application also provide a computer program product, including a non-volatile computer-readable storage medium for storing a computer program, which is executed by a processor to implement the methods in various embodiments of this application.

[0140] Embodiments of this application also provide a computer program that, when run by a processor, implements the methods described in the various embodiments of this application.

[0141] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not operational. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0145] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to run all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0146] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data annotation method, characterized in that, include: Multimodal data is obtained by acquiring modal data in at least one mode from multiple sensors deployed in the vehicle, wherein the multimodal data is used to characterize the traffic state when the vehicle is located in a traffic area, and the traffic area is within a target area. The multimodal data is input into the annotation model for annotation to obtain at least one initial annotation result. The annotation model is trained based on the multimodal data samples collected by the sensor and the annotation result samples corresponding to the multimodal data samples. The initial annotation result is used to characterize the location of at least one obstacle in the traffic area. In response to the initial annotation result having a confidence level lower than a confidence threshold, an adjustment instruction is obtained; In response to the adjustment instruction, the three-dimensional spatial labels of the obstacles in the initial annotation results are adjusted to obtain the target annotation results, wherein the three-dimensional spatial labels are used to characterize the location and size of the obstacles.

2. The method according to claim 1, characterized in that, The step of inputting the multimodal data into the annotation model for annotation to obtain at least one initial annotation result includes: The multimodal data is time-aligned according to the timestamp; The time-aligned multimodal data is corrected to obtain multiple corrected data. Multiple sets of correction data are input into the annotation model for annotation, resulting in at least one initial annotation result.

3. The method according to claim 2, characterized in that, The step of inputting multiple sets of correction data into the annotation model for annotation to obtain at least one initial annotation result includes: The prompt information and multiple correction data are input into the annotation model to obtain at least one initial annotation result, wherein the prompt information is used to prompt the annotation model for the obstacle to be annotated.

4. The method according to claim 1, characterized in that, The response to the initial annotation result having a confidence level lower than a confidence level threshold, obtaining an adjustment instruction, includes: In response to the initial annotation result having a confidence level lower than the confidence threshold, the initial annotation result is annotated in at least one of the modal data corresponding to the initial annotation result; The labeled modal data is displayed on the display interface; Obtain the operation data from the display interface; The operation data is converted into the adjustment instructions.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: At multiple consecutive time points, the similarity between multiple target annotation results corresponding to the obstacle is determined, wherein the target annotation results correspond one-to-one with the time points; In response to the similarity being lower than a similarity threshold, the obstacle is marked and then re-labeled.

6. The method according to any one of claims 1 to 4, characterized in that, The method further includes: In response to the initial annotation result having a confidence level lower than the confidence level threshold, the multimodal data is used as the multimodal data sample, and the initial annotation result is used as the annotation result sample, and stored in the database; In response to the number of multimodal data samples in the database exceeding a data volume threshold, a target number of multimodal data samples and the corresponding labeled result samples are selected from the database. The annotation model is updated using the selected multimodal data samples and the corresponding annotation result samples.

7. A data annotation device, characterized in that, include: The first acquisition unit is used to acquire modal data in at least one mode collected by multiple sensors deployed in the vehicle to obtain multimodal data, wherein the multimodal data is used to characterize the traffic state when the vehicle is located in a traffic area, and the traffic area is within the target area. The annotation unit is used to input the multimodal data into the annotation model for annotation to obtain at least one initial annotation result. The annotation model is trained based on the multimodal data samples collected by the sensor and the annotation result samples corresponding to the multimodal data samples. The initial annotation result is used to characterize the location of at least one obstacle in the traffic area. The second acquisition unit is used to acquire an adjustment instruction in response to the confidence level of the initial annotation result being lower than the confidence threshold. An adjustment unit is configured to adjust the three-dimensional spatial labels of the obstacles in the initial annotation result in response to the adjustment command, thereby obtaining the target annotation result, wherein the three-dimensional spatial labels are used to characterize the position and size of the obstacles.

8. A vehicle, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the storage medium is located to perform the method of any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the method described in any one of claims 1 to 6.

Images