Multi-modal data labeling method and device and storage medium

Abstract

The invention provides a multi-modal data labeling method and device and a storage medium, and relates to the field of automatic driving, and the method comprises the steps: synchronously collecting multi-modal data, and obtaining environment parameters; based on the environmental parameters, obtaining the reliability of each modal, judging the modal with the highest modal reliability as a reliable modal, and judging other modals as unreliable modals; the pre-labeling result of the reliable mode is a final labeling result of the reliable mode; transforming the pre-labeling result of the reliable mode to obtain a projection result of the unreliable mode, wherein the projection result of the pre-labeling result of the reliable mode is projected on the unreliable mode; inputting the projection result of the unreliable mode and the pre-labeling result of the unreliable mode into a target association module, and outputting an intersection-to-union ratio; and correcting the pre-labeling result of the unreliable mode according to the intersection-parallel comparison to obtain a final labeling result of the unreliable mode. The method effectively solves the problems of poor environmental adaptability and insufficient multi-modal data fusion precision in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, in particular to a multi-modal data labeling method, device and storage medium. BACKGROUND

[0002] With the rapid development of automatic driving technology, multi-modal perception systems have become the core technical support for comprehensive environmental understanding. Modern autonomous vehicles are usually equipped with visible light cameras, lidar, infrared cameras, millimeter wave radars and other sensors, which can provide complementary scene information under different environmental conditions. Visible light cameras can capture rich color and texture features, lidar provides accurate three-dimensional spatial structure, infrared cameras and millimeter wave radars have the ability to penetrate rain and fog and are not affected by light conditions, together forming the basis of all-weather, all-scene perception capabilities.

[0003] In the prior art, some multi-modal large model-based data labeling methods have been proposed. These methods usually detect sensor data through multi-modal large models, automatically label targets that meet preset conditions, and further label data scenes according to target labeling results, and finally filter and mine data based on labeling results. This kind of technology mainly targets the labeling process of lidar point cloud and visible light image, and provides certain automated data labeling capability.

[0004] However, the prior art has obvious limitations. First, these methods do not fully consider the pre-labeling needs of infrared image data, and cannot effectively handle the low contrast and texture blur characteristics unique to infrared images, limiting their application range in complete multi-modal perception systems. Second, the prior art lacks a correction mechanism based on pre-labeling results, and does not consider the problems of temporal and spatial synchronization errors and calibration parameter deviations in the actual acquisition process of multi-modal sensors, resulting in a significant impact on labeling accuracy and consistency when fusing multi-modal data, making it difficult to meet the strict requirements of automatic driving systems for high-precision labeled data. SUMMARY

[0005] The present application aims to at least solve the above-mentioned technical problems in the prior art. To this end, the first aspect of the present application proposes a multi-modal data labeling method, the method comprising:

[0006] synchronously collecting multi-modal data and obtaining environmental parameters; the multi-modal data includes lidar point cloud data, visible light camera images, infrared camera images, and the environmental parameters include light intensity, visibility, precipitation intensity, environmental temperature and relative humidity;

[0007] based on the environmental parameters, obtaining the reliability of each modality, judging the modality with the highest modality reliability as the reliable modality, and judging the other modalities as non-reliable modalities.

[0008] The multimodal data is pre-labeled based on the large object detection model to obtain multimodal pre-labeling results; the multimodal pre-labeling results include the pre-labeling results of reliable modes and the pre-labeling results of unreliable modes; the pre-labeling results of reliable modes are the final labeling results;

[0009] The pre-labeling results of the reliable modes are transformed to obtain the projection results of the reliable modes onto the unreliable modes.

[0010] The projection results of the unreliable modes and the pre-labeling results of the unreliable modes are input into the target association module, and the intersection-union ratio is output.

[0011] The pre-labeling results of the unreliable modes are corrected based on the intersection-union comparison to obtain the final labeling results of the unreliable modes.

[0012] Optionally, the modal reliability includes lidar reliability, visible light camera reliability, and infrared camera reliability. The lidar reliability is determined based on visibility, precipitation intensity, ambient temperature, and relative humidity. The visible light camera reliability is determined based on light intensity, visibility, precipitation intensity, and relative humidity. The infrared camera reliability is determined based on precipitation intensity, ambient temperature, and relative humidity.

[0013] Optionally, the pre-labeling results of unreliable modalities are corrected based on the intersection-union ratio (IU), including: if the IU is greater than a first threshold, the pre-labeling results of the unreliable modalities are output as the final labeling results; if the IU is less than the first threshold but greater than a second threshold, a weighted labeling result of the unreliable modalities is determined based on the projection labeling results and the pre-labeling results of the unreliable modalities, and the weighted labeling result of the unreliable modalities is used as the final labeling result; if the IU is less than the second threshold, the target detection large model is called for secondary labeling, the secondary labeling result of the unreliable modalities is output, and the secondary labeling result of the unreliable modalities is used as the final labeling result.

[0014] Optionally, the pre-labeling results of the reliable mode are transformed, including: when the pre-labeling results of the reliable mode are point cloud data labels, the three-dimensional point cloud targets are mapped to the coordinate system of the unreliable mode through projection transformation; when the pre-labeling results of the reliable mode are image data labels, the two-dimensional image labels are converted into three-dimensional spatial information using inverse projection technology.

[0015] Optionally, the weighted annotation result of the unreliable mode includes: determining the weight coefficient based on the mode reliability, and performing weighted fusion of the bounding box coordinates of the projected annotation result and the pre-annotation result to obtain the weighted annotation result.

[0016] Optionally, the step of calling the target detection large model for secondary annotation includes: expanding the detection area with the bounding box of the projection annotation result or the pre-annotation result as the center, adjusting the detection threshold, and then re-performing the target detection.

[0017] Optionally, the method further includes:

[0018] Based on the predefined labeling difficulty grading standard, the final labeling results of the unreliable modal are stratified and sampled. For the labeled data classified as high difficulty level, a manual verification and correction process is initiated to obtain the verified labeling results.

[0019] Optionally, the method further includes:

[0020] The verified annotation results are stored in a dedicated database and standardized to be compatible with different data interface requirements.

[0021] A second aspect of the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the multimodal data annotation method as proposed in the first aspect.

[0022] A third aspect of the present invention provides a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the multimodal data annotation method as proposed in the first aspect.

[0023] The beneficial effects of a method, device, and storage medium for labeling multimodal data are:

[0024] This invention effectively addresses the lack of infrared data annotation in existing technologies by using a large-scale target detection model to uniformly pre-annotate multimodal data, including infrared images. It can handle the unique characteristics of infrared images, such as low contrast and blurred textures. An environmental parameter analysis module evaluates the reliability of each sensor in real time, dynamically selecting the optimal mode based on environmental parameters such as light intensity and visibility, overcoming the performance limitations of single modes under specific environmental conditions. Through cross-modal annotation mapping, combined with cross-intersection over union (CUI) analysis and a multi-level decision-making mechanism in the target association module, it effectively eliminates spatiotemporal synchronization errors and calibration parameter deviations among multimodal sensors. Weighted fusion and secondary annotation mechanisms improve annotation quality while ensuring annotation efficiency. Secondary annotation is automatically initiated when there is a significant deviation in the pre-annotation results, ensuring the accuracy and reliability of the final annotation results. This invention effectively solves the technical problems of poor environmental adaptability and insufficient multimodal data fusion accuracy in existing technologies through the synergistic effect of environmentally perceptive mode selection and multi-level annotation correction. Attached Figure Description

[0025] Figure 1 A flowchart of a multimodal data annotation method provided in this embodiment of the invention. Figure 1 ;

[0026] Figure 2 A flowchart of a multimodal data annotation method provided in this embodiment of the invention. Figure 2 ;

[0027] Figure 3 A flowchart of the annotation correction module provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, the use of "based on" or "according to" implies openness and inclusiveness, because processes, steps, calculations, or other actions "based on" or "according to" one or more of the stated conditions or values ​​may in practice be based on additional conditions or beyond the stated values.

[0030] This invention provides a method for labeling multimodal data, such as... Figure 1 and Figure 2 As shown, the method may include the following steps:

[0031] Step 101: Synchronously collect multimodal data and obtain environmental parameters.

[0032] The multimodal data includes lidar point cloud data, visible light camera images, and infrared camera images, while the environmental parameters include light intensity, visibility, precipitation intensity, ambient temperature, and relative humidity.

[0033] Specifically, a hardware platform for multimodal data acquisition is built, and a sensor array is integrated and installed on the data acquisition vehicle. The sensor array includes a lidar sensor, a visible light camera, and an infrared camera. The lidar sensor is a 128-line mechanical rotating lidar with a sampling frequency of 10Hz, a horizontal field of view of 360°, and a vertical field of view of 30°. The visible light camera is an 8-megapixel global shutter camera with a frame rate of 25fps, equipped with a 6mm fixed-focus lens, covering a horizontal field of view of 75°. The infrared camera uses a 640×512 resolution uncooled infrared focal plane array with a spectral range of 8-14μm, a noise equivalent temperature difference of less than 50mK, and a frame rate of 25Hz.

[0034] To achieve accurate multi-sensor data fusion, time alignment and spatial calibration are performed on each sensor. Time alignment is based on the PTP precision clock protocol to timestamp and synchronize the data streams of each sensor. Spatial calibration utilizes pre-calibrated sensor extrinsic parameter matrices to achieve spatial alignment between the LiDAR and the visible light camera, the LiDAR and the infrared camera, and the infrared camera and the visible light camera. Each sensor achieves microsecond-level time synchronization through a hardware synchronization trigger device, with synchronization errors controlled within 1ms.

[0035] Specifically, the collected raw data is preprocessed, including using lossless compression algorithms to reduce storage space usage and establishing an index relationship between environmental parameters and each frame of data to form a complete multimodal dataset.

[0036] Step 102: Based on environmental parameters, obtain the reliability of each mode, and determine the mode with the highest modal reliability as the reliable mode, and the other modes as unreliable modes.

[0037] Specifically, the method for pre-annotating multimodal data is as follows: deploy a parallel annotation framework based on a large model, load and run pre-trained 3D and 2D target detection models, and pre-annotate targets for LiDAR point cloud data, visible light camera images, and infrared camera images respectively, forming three parallel processing branches:

[0038] The point cloud annotation branch employs a VoxelNext-improved sparse convolutional network architecture. The input data is a 128-line LiDAR point cloud, with each frame containing approximately 200,000 point cloud data points. This branch outputs complete 3D target annotation information, including the center coordinates (x, y, z) of the 3D bounding box, geometric parameters such as length, width, and height, and yaw angle, as well as semantic category annotations for seven types of vehicles, including cars, pedestrians, bicycles, and tricycles. It also assigns a cross-frame tracking instance ID to each detected target. This branch achieves an mAP@0.5 of 85.3% on the KITTI dataset.

[0039] The visible light image annotation branch is built on the Swing Transformer architecture, taking 1920×1080 resolution RGB images as input. This branch outputs the coordinates of the top-left and bottom-right corners of the 2D bounding box, semantic category labels aligned with the point cloud category, and a confidence score in the range of 0-1. Evaluation results on the COCO dataset show that this branch achieves an mAP of 78.6%.

[0040] The infrared image annotation branch employs an improved YOLOv10 architecture, with input data consisting of 640×512 resolution thermal images. This branch outputs 2D bounding box coordinates, semantic category labels consistent with the point cloud categories, and confidence scores. Performance evaluation on a self-built infrared dataset shows that this branch achieves a PR-AUC of 0.823.

[0041] In one possible implementation, the reliability of the multimodal data includes the reliability of lidar, visible light camera, and infrared camera. The lidar reliability is determined based on visibility, precipitation intensity, ambient temperature, and relative humidity. The visible light camera reliability is determined based on light intensity, visibility, precipitation intensity, and relative humidity. The infrared camera reliability is determined based on precipitation intensity, ambient temperature, and relative humidity.

[0042] LiDAR reliability The calculation uses the following formula:

[0043]

[0044]

[0045]

[0046]

[0047] in, The visibility penalty is denoted by v, where v is the visibility. Humidity penalty, where h is the humidity; Penalty for insufficient rainfall; T is the temperature factor, representing temperature.

[0048] Visible light camera reliability The calculation uses the following formula:

[0049]

[0050]

[0051] ,

[0052]

[0053] in, Let L be the illumination factor and L be the illumination intensity.

[0054] Infrared camera reliability The calculation uses the following formula:

[0055]

[0056]

[0057]

[0058] in, This is the temperature contrast factor.

[0059] Rainfall penalty The precipitation intensity is determined according to the rules in Table 1.

[0060] Table 1

[0061] Precipitation intensity Lidar penalty value Visible camera penalty value Infrared camera penalty value No precipitation 0 0 0 Light rain 0.2 0.3 0.1 Moderate rain 0.4 0.6 0.3 Heavy rain 0.6 0.8 0.5 Torrential rain 0.8 0.9 0.7

[0062] Step 103: Transform the pre-labeling results of the reliable modes to obtain the projection results of the reliable modes onto the unreliable modes.

[0063] In one possible implementation, the pre-labeling results of the unreliable mode are corrected based on the intersection-union comparison, including: when the pre-labeling result of the reliable mode is point cloud data labeling, the three-dimensional point cloud target is mapped to the coordinate system of the unreliable mode through projection transformation; when the pre-labeling result of the reliable mode is image data labeling, the two-dimensional image labeling is converted into three-dimensional spatial information using inverse projection technology.

[0064] Specifically, the pre-annotation results of the modalities are transformed using a projection transformation module. The projection transformation method is as follows:

[0065] When the pre-annotation result of the reliable mode is point cloud data annotation, the three-dimensional bounding box detected by the lidar is converted into a two-dimensional bounding box in the image coordinate system of the unreliable mode by using the pre-calibrated spatial transformation relationship between sensors. These projection results are used as the projection annotation results of the unreliable mode.

[0066] When the pre-labeling result of the reliable modality is image data labeling, the camera height and pitch angle are obtained through camera calibration parameters. The pixel coordinates of the bottom midpoint of the detection box are back-projected into the camera coordinate system according to the horizon assumption. Then, the points in the camera coordinate system are transformed into the point cloud coordinate system. Finally, all point clouds belonging to the target are located through clustering methods to obtain the corresponding 3D annotation information.

[0067] Through the spatial projection processing described above, the spatial projection module can establish accurate annotation correspondences between different modalities, providing a reliable foundation for subsequent target association and annotation correction. The sensor extrinsic parameter matrix used during the projection process remains consistent with the calibration results from the data acquisition phase, ensuring the accuracy of the spatial transformation.

[0068] Step 104: Input the projection results of the unreliable modes and the pre-annotation results of the unreliable modes into the target association module, and output the intersection-union ratio.

[0069] Specifically, the projection results of unreliable modes and the pre-annotation results of unreliable modes are input into the intersection-union-ratio (IUGR) correction module. The IUGR correction module includes a target association module, such as... Figure 3 As shown, the matching problem between the projection annotation results and the pre-annotation results of the unreliable modality is handled by the target association module; the Hungarian algorithm is used for optimal bipartite graph matching, and the intersection-union ratio is used as the matching metric to establish the association relationship, ensuring that the target in each reliable modality corresponds to at most one unreliable detection result; the pre-annotation complexity of this frame data is defined as the average intersection-union ratio of all successfully matched pairs.

[0070] Step 105: Correct the pre-labeled results of the unreliable modes based on the intersection-union comparison to obtain the final labeling results of the unreliable modes.

[0071] In one possible implementation, corrections are made based on the pre-labeled results of the cross-union comparison of unreliable modes, including:

[0072] If the intersection-union ratio (IU) is greater than the first threshold, the pre-labeled result of the unreliable mode is output as the final labeling result; if the IU is less than the first threshold but greater than the second threshold, the weighted labeling result of the unreliable mode is determined based on the projection labeling result of the unreliable mode and the pre-labeling result of the unreliable mode, and the weighted labeling result of the unreliable mode is used as the final labeling result; if the IU is less than the second threshold, the large object detection model is called for secondary labeling, the secondary labeling result of the unreliable mode is output, and the secondary labeling result of the unreliable mode is used as the final labeling result.

[0073] Specifically, the first threshold can be set to 0.7, and the second threshold can be set to 0.3.

[0074] like Figure 3 As shown, based on the cross-union ratio (CUR) value output by the target association module, a hierarchical processing strategy is adopted: when the CUR is greater than 0.7, the detection results of the two modalities are considered to be highly consistent, and the pre-labeled results of the unreliable modality are directly adopted; when the CUR is between 0.3 and 0.7, the frame fusion algorithm based on reliability weighting is used to generate the final labeling results; when the CUR is less than 0.3, it is identified as a significant difference and further analysis and processing are required.

[0075] In one possible implementation, the weighted annotation result of the unreliable mode includes: determining the weight coefficients based on the modal reliability, and performing weighted fusion of the bounding box coordinates of the projected annotation result and the pre-annotated result to obtain the weighted annotation result.

[0076] Specifically, for target pairs that are successfully matched and have an intersection-union ratio between 0.3 and 0.7, a reliability-weighted frame fusion algorithm is used to generate the final annotation result. The specific fusion formula is as follows:

[0077]

[0078] in, This represents the coordinates of the final bounding box after merging. The coordinates of the projected bounding box representing the reliable mode. The coordinates of the pre-labeled bounding box representing the unreliable mode;

[0079] The formula for calculating the weighting coefficient α is: ,in This represents the reliability score of the reliable modes under current environmental conditions. This represents the reliability score of the unreliable mode under the current environmental conditions.

[0080] In one possible implementation, a large object detection model is invoked for secondary annotation, including: expanding the detection area centered on the bounding box of the projected annotation result or the pre-annotated result, adjusting the detection threshold, and then re-performing object detection.

[0081] Specifically, for significant differences with an intersection-union ratio less than 0.3, the following two scenarios are handled:

[0082] When a reliable mode detects a target but an unreliable mode fails to detect it, a target detection region with a scaled-up ratio of 2 is constructed based on the center of the projection region of the reliable mode onto the unreliable mode. Within this region, the detection threshold is lowered, and large-scale pre-annotation is performed again. If a valid detection result is obtained, it is confirmed that the target was missed by the unreliable mode, and the projection result is used as the final annotation. If no valid detection result is obtained, it is confirmed that the inconsistency is caused by occlusion, and the target is excluded from the unreliable mode.

[0083] When an unreliable mode is detected but a reliable mode is not, a target detection region with a scale increased by two times is constructed based on the center of the unreliable mode detection region. Within this region, the detection threshold is increased, and large-scale model pre-labeling is performed again. If valid detection results are still obtained, the inconsistency is confirmed to be caused by occlusion, and the unreliable mode detection results are used as the final labeling. If no valid detection results are obtained, the unreliable mode is confirmed as a false detection, and the target is excluded from the final result.

[0084] In summary, this invention utilizes a large-scale target detection model to uniformly pre-label multimodal data, including infrared images, effectively addressing the lack of infrared data labeling in existing technologies and handling the unique characteristics of infrared images, such as low contrast and blurred textures. An environmental parameter analysis module evaluates the reliability of each sensor in real time, dynamically selecting the optimal modality based on environmental parameters such as light intensity and visibility, overcoming the performance limitations of a single modality under specific environmental conditions. A spatial projection module enables cross-modal labeling mapping, and combined with the intersection-union analysis and multi-level decision-making mechanism of the target association module, effectively eliminating spatiotemporal synchronization errors and calibration parameter deviations among multimodal sensors. Through weighted fusion and secondary labeling mechanisms, labeling efficiency is ensured while labeling quality is improved. Secondary labeling is automatically initiated when there is a significant deviation in the pre-labeling results, ensuring the accuracy and reliability of the final labeling results. This invention effectively solves the technical problems of poor environmental adaptability and insufficient multimodal data fusion accuracy in existing technologies through the synergistic effect of environmentally perceptive modality selection and multi-level labeling correction.

[0085] In one possible implementation, the corrected multimodal annotation results are stratified and sampled according to a predefined annotation difficulty grading standard. For the annotated data classified as high difficulty level, a manual verification and correction process is initiated to obtain the verified annotation results.

[0086] This verification mechanism, through human-machine collaboration, focuses on reviewing potential difficult cases and boundary situations that may exist during the automatic annotation process. This not only effectively ensures the accuracy and consistency of the annotation results but also optimizes the allocation of human review resources.

[0087] The manual verification process serves as an effective supplement to the automatic pre-annotation of large models. It can specifically identify and correct potential minor errors in highly complex samples, especially those complex scenarios that might have been missed or misjudged during the pre-annotation result correction step. Through this quality verification strategy based on difficulty stratification, the system achieves an optimal balance between annotation cost and benefit while ensuring annotation quality.

[0088] In one possible implementation, the verified annotation results are stored in a dedicated database and standardized to be compatible with different data interface requirements.

[0089] This database provides data support for various subsequent application scenarios, including but not limited to the training of autonomous driving perception models and the development of target recognition and avoidance algorithms for decision-making systems.

[0090] During the standardization process, the validated annotation results are converted into industry-standard data formats. For example, the 3D annotation results of LiDAR point clouds are converted into the KITTI standard format, while the annotation results of image data are converted into common 2D target detection formats such as Pascal VOC or COCO, depending on the application requirements. This standardization process ensures good compatibility between the generated data and mainstream algorithm frameworks and training processes, greatly enhancing the practical value and ease of sharing of the data.

[0091] In another embodiment of the present invention, an electronic device is also provided, the electronic device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the multimodal data annotation method proposed in the embodiments of the present invention.

[0092] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or at least one program is loaded and executed by the processor to implement the multimodal data annotation method proposed in the embodiments of the present invention.

[0093] The foregoing primarily describes the solutions provided by the embodiments of the present invention from the perspective of the device. It is understood that, in order to achieve the above functions, the device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the algorithmic steps of the various examples described in the embodiments disclosed herein, the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0094] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for labeling multimodal data, characterized in that, include: Simultaneously collect multimodal data and acquire environmental parameters; the multimodal data includes lidar point cloud data, visible light camera images, and infrared camera images; the environmental parameters include light intensity, visibility, precipitation intensity, ambient temperature, and relative humidity; Based on the environmental parameters, the reliability of each mode is obtained, and the mode with the highest modal reliability is judged as a reliable mode, while other modes are judged as unreliable modes. The multimodal data is pre-labeled based on the large object detection model to obtain multimodal pre-labeling results; the multimodal pre-labeling results include the pre-labeling results of reliable modes and the pre-labeling results of unreliable modes; the pre-labeling results of reliable modes are the final labeling results of reliable modes; The pre-labeling results of the reliable modes are transformed to obtain the projection results of the pre-labeling results of the reliable modes onto the unreliable modes; The projection results on the unreliable modes and the pre-labeling results of the unreliable modes are input into the target association module, and the intersection-union ratio is output. The pre-labeling results of the unreliable modes are corrected based on the intersection-union comparison to obtain the final labeling results of the unreliable modes.

2. The method for labeling multimodal data according to claim 1, characterized in that, The modal reliability includes the reliability of lidar, visible light camera, and infrared camera; the lidar reliability is determined based on visibility, precipitation intensity, ambient temperature, and relative humidity; the visible light camera reliability is determined based on light intensity, visibility, precipitation intensity, and relative humidity; and the infrared camera reliability is determined based on precipitation intensity, ambient temperature, and relative humidity.

3. The method for labeling multimodal data according to claim 1, characterized in that, The step of correcting the pre-labeled results of unreliable modes based on the intersection-union comparison includes: If the intersection-union ratio (IU) is greater than a first threshold, the pre-labeling result of the unreliable mode is output as the final labeling result; if the IU is less than the first threshold but greater than a second threshold, the weighted labeling result of the unreliable mode is determined based on the projection labeling result and the pre-labeling result of the unreliable mode, and the weighted labeling result of the unreliable mode is used as the final labeling result; if the IU is less than the second threshold, the target detection large model is called for secondary labeling, the secondary labeling result of the unreliable mode is output, and the secondary labeling result of the unreliable mode is used as the final labeling result.

4. The method for labeling multimodal data according to claim 1, characterized in that, The transformation of the pre-labeled results of the reliable modes includes: When the pre-annotation result of the reliable mode is point cloud data annotation, the three-dimensional point cloud target is mapped to the coordinate system of the unreliable mode through projection transformation; when the pre-annotation result of the reliable mode is image data annotation, the two-dimensional image annotation is converted into three-dimensional spatial information using inverse projection technology.

5. The method for labeling multimodal data according to claim 3, characterized in that, The weighted labeling results of the unreliable modes include: The weighting coefficients are determined based on the modal reliability, and the bounding box coordinates of the projection annotation results and the pre-annotation results are weighted and fused to obtain the weighted annotation results.

6. The method for labeling multimodal data according to claim 3, characterized in that, The process of calling the large object detection model for secondary annotation includes: Expand the detection area centered on the bounding box of the projection annotation result or pre-annotation result, adjust the detection threshold, and then re-perform target detection.

7. The method for labeling multimodal data according to claim 1, characterized in that, The method further includes: Based on the predefined annotation difficulty grading standard, the final annotation results of the obtained unreliable modalities are subjected to stratified sampling. For the annotation data classified as high difficulty level, a manual verification and correction process is initiated to obtain the verified annotation results.

8. The method for labeling multimodal data according to claim 7, characterized in that, The method further includes: The verified annotation results are stored in a dedicated database and standardized to be compatible with different data interface requirements.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the multimodal data annotation method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the multimodal data annotation method as described in any one of claims 1-8.

Images