A method and device for labeling multi-modal data, and a storage medium

Abstract

The application provides a multi-modal data labeling method and device and a storage medium, and relates to the field of automatic driving. The method comprises the following steps: synchronously collecting multi-modal data and obtaining environment parameters; based on the environment parameters, obtaining modal reliabilities, judging a modal with the highest modal reliability as a reliable modal, and judging other modals as unreliable modals; a pre-labeling result of the reliable modal is a final labeling result of the reliable modal; the pre-labeling result of the reliable modal is transformed to obtain a projection result of the pre-labeling result of the reliable modal on the unreliable modals; the projection result of the reliable modals and the pre-labeling result of the unreliable modals are input into a target association module to output an intersection-over-union ratio; and the pre-labeling result of the unreliable modals is corrected according to the intersection-over-union ratio to obtain a final labeling result of the unreliable modals. The method effectively solves the problems of poor environment adaptability and insufficient multi-modal data fusion precision in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving, in particular to a multi-modal data labeling method, device and storage medium. BACKGROUND

[0002] With the rapid development of automatic driving technology, multi-modal perception systems have become the core technical support for comprehensive environmental understanding. Modern autonomous vehicles are usually equipped with visible light cameras, lidar, infrared cameras, millimeter wave radars and other sensors, which can provide complementary scene information under different environmental conditions. Visible light cameras can capture rich color and texture features, lidar provides accurate three-dimensional spatial structure, infrared cameras and millimeter wave radars have the ability to penetrate rain and fog and are not affected by light conditions, together forming the basis of all-weather, all-scene perception capabilities.

[0003] In the prior art, some multi-modal large model-based data labeling methods have been proposed. These methods usually detect sensor data through multi-modal large models, automatically label targets that meet preset conditions, and further label data scenes according to target labeling results, and finally filter and mine data based on labeling results. This kind of technology mainly targets the labeling process of lidar point cloud and visible light image, and provides certain automated data labeling capability.

[0004] However, the prior art has obvious limitations. First, these methods do not fully consider the pre-labeling needs of infrared image data, and cannot effectively handle the low contrast and texture blur characteristics unique to infrared images, limiting their application range in complete multi-modal perception systems. Second, the prior art lacks a correction mechanism based on pre-labeling results, and does not consider the problems of temporal and spatial synchronization errors and calibration parameter deviations in the actual acquisition process of multi-modal sensors, resulting in a significant impact on labeling accuracy and consistency when fusing multi-modal data, making it difficult to meet the strict requirements of automatic driving systems for high-precision labeled data. SUMMARY

[0005] The present application aims to at least solve the above-mentioned technical problems in the prior art. To this end, the first aspect of the present application proposes a multi-modal data labeling method, the method comprising:

[0006] synchronously collecting multi-modal data and obtaining environmental parameters; the multi-modal data includes lidar point cloud data, visible light camera images, infrared camera images, and the environmental parameters include light intensity, visibility, precipitation intensity, environmental temperature and relative humidity;

[0007] based on the environmental parameters, obtaining the reliability of each modality, judging the modality with the highest modality reliability as the reliable modality, and judging the other modalities as non-reliable modalities.

[0008] perform pre-labeling on the multi-modal data based on the target detection large model to obtain a pre-labeling result of the multi-modal data; the pre-labeling result of the multi-modal data includes a pre-labeling result of a reliable modal and a pre-labeling result of an unreliable modal; the pre-labeling result of the reliable modal is a final labeling result;

[0009] transform the pre-labeling result of the reliable modal to obtain a projection result of the pre-labeling result of the reliable modal on the unreliable modal;

[0010] input the projection result of the unreliable modal and the pre-labeling result of the unreliable modal into a target association module to output an intersection over union;

[0011] correct the pre-labeling result of the unreliable modal according to the intersection over union to obtain a final labeling result of the unreliable modal.

[0012] Optionally, the modal reliability includes laser radar reliability, visible light camera reliability, and infrared camera reliability; the laser radar reliability is determined based on visibility, precipitation intensity, ambient temperature, and relative humidity; the visible light camera reliability is determined based on illumination intensity, visibility, precipitation intensity, and relative humidity; and the infrared camera reliability is determined based on precipitation intensity, ambient temperature, and relative humidity.

[0013] Optionally, correcting the pre-labeling result of the unreliable modal according to the intersection over union includes: if the intersection over union is greater than a first threshold, outputting the pre-labeling result of the unreliable modal as the final labeling result; if the intersection over union is less than the first threshold and greater than a second threshold, determining a weighted labeling result of the unreliable modal based on the projection labeling result of the unreliable modal and the pre-labeling result of the unreliable modal, and outputting the weighted labeling result of the unreliable modal as the final labeling result; and if the intersection over union is less than the second threshold, calling the target detection large model to perform secondary labeling, and outputting a secondary labeling result of the unreliable modal, and outputting the secondary labeling result of the unreliable modal as the final labeling result.

[0014] Optionally, transforming the pre-labeling result of the reliable modal includes: when the pre-labeling result of the reliable modal is point cloud data labeling, mapping a three-dimensional point cloud target into a coordinate system of the unreliable modal through projection transformation; and when the pre-labeling result of the reliable modal is image data labeling, converting two-dimensional image labeling into three-dimensional space information by using inverse projection technology.

[0015] Optionally, the weighted labeling result of the unreliable modal includes: determining a weight coefficient according to the modal reliability, and performing weighted fusion on bounding box coordinates of the projection labeling result and the pre-labeling result to obtain the weighted labeling result.

[0016] Optionally, the calling target detection large model for secondary labeling comprises: expanding the detection area with the boundary box of the projection labeling result or the pre-labeling result as the center, and readjusting the detection threshold to re-perform target detection.

[0017] Optionally, the method further comprises:

[0018] According to the pre-defined labeling difficulty grading standard, the final labeling result of the non-reliable modality is stratified sampled, for the labeling data divided into a high difficulty level, an artificial verification and correction process is started to obtain a verified labeling result.

[0019] Optionally, the method further comprises:

[0020] The verified labeling result is stored in a special database and is standardized to be compatible with different data interface requirements.

[0021] The second aspect of the application provides an electronic device, comprising a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or at least one program is loaded and executed by the processor to realize the labeling method of multi-modal data as the first aspect.

[0022] The third aspect of the application provides a computer readable storage medium, the storage medium stores at least one instruction or at least one program, the at least one instruction or at least one program is loaded and executed by the processor to realize the labeling method of multi-modal data as the first aspect.

[0023] The labeling method of multi-modal data, the device and the storage medium have the following beneficial effects:

[0024] The target detection large model is used for unified pre-labeling of multi-modal data containing infrared images, effectively solving the problem of missing infrared data labeling in the prior art, and being capable of processing the low contrast and texture blur characteristics unique to infrared images; the environmental parameter analysis module is used for real-time evaluation of the reliability of each sensor, and the optimal modality is dynamically selected based on environmental parameters such as illumination intensity and visibility, overcoming the performance limitations of a single modality under specific environmental conditions; through cross-modal labeling mapping, combined with the intersection over union analysis and multi-level decision mechanism of the target association module, the temporal and spatial synchronization errors and calibration parameter deviations between multi-modal sensors are effectively eliminated; through weighted fusion and secondary labeling mechanism, the labeling efficiency is ensured while the labeling quality is improved, and when the pre-labeling result has a large deviation, secondary labeling is automatically started to ensure the accuracy and reliability of the final labeling result. Through the synergistic effect of environmental perception modality selection and multi-level labeling correction, the application effectively solves the technical problems of poor environmental adaptability and insufficient multi-modal data fusion accuracy in the prior art. Attached Figure Description

[0025] Figure 1 A flowchart of a multimodal data annotation method provided in this embodiment of the invention. Figure 1 ;

[0026] Figure 2 A flowchart of a multimodal data annotation method provided in this embodiment of the invention. Figure 2 ;

[0027] Figure 3 A flowchart of the annotation correction module provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, the use of "based on" or "according to" implies openness and inclusiveness, because processes, steps, calculations, or other actions "based on" or "according to" one or more of the stated conditions or values ​​may in practice be based on additional conditions or beyond the stated values.

[0030] This invention provides a method for labeling multimodal data, such as... Figure 1 and Figure 2 As shown, the method may include the following steps:

[0031] Step 101: Synchronously collect multimodal data and obtain environmental parameters.

[0032] The multimodal data includes lidar point cloud data, visible light camera images, and infrared camera images, while the environmental parameters include light intensity, visibility, precipitation intensity, ambient temperature, and relative humidity.

[0033] Specifically, a hardware platform for multi-modal data collection is built, and a sensor array is integrated on a data collection vehicle, including a laser radar sensor, a visible light camera, and an infrared camera. The laser radar sensor adopts a 128-line mechanical rotating laser radar, with a collection frequency of 10 Hz, a horizontal field of view of 360°, and a vertical field of view of 30°. The visible light camera adopts an 8 million pixel global shutter camera, with a frame rate of 25 fps, equipped with a 6 mm fixed focus lens, covering a horizontal field of view of 75°. The infrared camera adopts a 640*512 resolution uncooled infrared focal plane array, with a spectral range of 8-14 μm, a noise equivalent temperature difference less than 50 mK, and a frame rate of 25 Hz.

[0034] To achieve the accuracy of multi-sensor data fusion, each sensor is time-aligned and spatially calibrated. Time alignment is based on PTP precise clock protocol to synchronize the time stamp of each sensor data stream; spatial calibration uses the pre-calibrated sensor extrinsic matrix to realize the spatial alignment between laser radar and visible light camera, laser radar and infrared camera, and infrared camera and visible light camera. Each sensor realizes microsecond-level time synchronization through a hardware synchronous trigger device, with a synchronization error controlled within 1 ms.

[0035] Specifically, the collected raw data is preprocessed, including using a lossless compression algorithm to reduce storage space occupation, and establishing an index relationship between environmental parameters and each frame of data to form a complete multi-modal data set.

[0036] Step 102, based on the environmental parameters, obtain the reliability of each modality, and determine the modality with the highest modality reliability as the reliable modality, and determine the other modalities as non-reliable modalities.

[0037] Specifically, the method for pre-annotating multi-modal data is: deploying a parallel annotation framework based on a large model, loading and running a pre-trained 3D and 2D target detection large model, and pre-annotating targets for laser radar point cloud data, visible light camera images, and infrared camera images respectively, forming three parallel processing branches:

[0038] The point cloud annotation branch adopts a sparse convolution network architecture improved by VoxelNext, and the input data is 128-line laser radar point cloud, containing about 200,000 point cloud data points per frame. This branch outputs complete 3D target annotation information, including the center coordinates (x, y, z) of the 3D bounding box, the length, width, and height dimensions, the yaw angle, and 7 semantic class labels such as cars, pedestrians, bicycles, and tricycles, and assigns an instance ID for each detection target for cross-frame tracking. The evaluation index of this branch on the KITTI dataset reaches mAP@0.5 of 85.3%.

[0039] The visible light image annotation branch is constructed based on the Swin Transformer architecture, and the input data is an RGB image with a resolution of 1920x1080. The branch outputs the upper left and lower right coordinates of the 2D bounding box, the semantic class label aligned with the point cloud class, and the confidence score in the range of 0-1. The evaluation results on the COCO dataset show that the mAP of this branch reaches 78.6%.

[0040] The infrared image annotation branch adopts an improved YOLOv10 architecture, and the input data is a thermal imaging image with a resolution of 640x512. The branch outputs 2D bounding box coordinates, semantic class labels consistent with point cloud classes, and confidence scores. Performance evaluation on the self-built infrared dataset shows that the PR-AUC of this branch reaches 0.823.

[0041] In one possible implementation, the reliability of the multi-modal data includes lidar reliability, visible light camera reliability, and infrared camera reliability, the lidar reliability is determined based on visibility, precipitation intensity, ambient temperature, and relative humidity, the visible light camera reliability is determined based on illumination intensity, visibility, precipitation intensity, and relative humidity, and the infrared camera reliability is determined based on precipitation intensity, ambient temperature, and relative humidity.

[0042] Lidar reliability The calculation uses the following formula:

[0043]

[0044]

[0045]

[0046]

[0047] wherein, is the visibility penalty, and v is the visibility; is the humidity penalty, and h is the humidity; is the precipitation amount penalty; is the temperature factor, and T represents the temperature.

[0048] Visible light camera reliability The calculation uses the following formula:

[0049]

[0050]

[0051] ,

[0052]

[0053] wherein, L is the light intensity.

[0054] Infrared camera reliability The calculation employs the following formula:

[0055]

[0056]

[0057]

[0058] wherein, T is the temperature contrast factor.

[0059] Precipitation penalty Determined according to the precipitation intensity according to Table 1 rules.

[0060] Table 1

[0061] Precipitation intensity Lidar penalty value Visible camera penalty value Infrared camera penalty value No precipitation 0 0 0 Light rain 0.2 0.3 0.1 Moderate rain 0.4 0.6 0.3 Heavy rain 0.6 0.8 0.5 Torrential rain 0.8 0.9 0.7

[0062] Step 103, transform the pre-annotation result of the reliable modality to obtain a projection result of the pre-annotation result of the reliable modality projected on the non-reliable modality of the non-reliable modality.

[0063] In one possible implementation, the pre-annotation result of the non-reliable modality is corrected according to the intersection over union, including: when the pre-annotation result of the reliable modality is point cloud data annotation, mapping a three-dimensional point cloud target to the coordinate system of the non-reliable modality through projection transformation; when the pre-annotation result of the reliable modality is image data annotation, converting two-dimensional image annotation to three-dimensional space information using inverse projection technology.

[0064] Specifically, the pre-annotation result of the reliable modality is transformed by a projection transformation module, and the projection transformation method is as follows:

[0065] When the pre-annotation result of the reliable modality is point cloud data annotation, the three-dimensional bounding box detected by the laser radar is converted into a two-dimensional bounding box in the image coordinate system of the non-reliable modality through projection calculation using the pre-calibrated spatial transformation relationship between sensors, and these projection results are used as the projection annotation result of the non-reliable modality;

[0066] When the pre-annotation result of the reliable modality is image data annotation, the camera height and pitch angle are obtained through camera calibration parameters, the bottom midpoint pixel coordinates of the detection frame are inversely projected into the camera coordinate system according to the ground assumption, and then the points in the camera coordinate system are converted into the point cloud coordinate system, and finally all the point clouds belonging to the target are located through clustering method to obtain the corresponding three-dimensional annotation information.

[0067] Through the above spatial projection processing, the spatial projection module can establish accurate annotation correspondence between different modalities, providing a reliable basis for subsequent target association and annotation correction. The sensor extrinsic matrix used in the projection process maintains consistency with the calibration results in the data acquisition stage, ensuring the accuracy of the spatial transformation.

[0068] Step 104, input the projection result of the unreliable modality and the pre-annotation result of the unreliable modality into the target association module, and output the intersection over union.

[0069] Specifically, the projection result of the unreliable modality and the pre-annotation result of the unreliable modality are input into the intersection over union correction module, which includes the target association module, as shown in Figure 3 The matching problem between the projection annotation result and the pre-annotation result of the unreliable modality is processed by the target association module; the Hungarian algorithm is used for optimal bipartite graph matching, and the intersection over union is used as the matching metric index to establish the association relationship, ensuring that each target in the reliable modality corresponds to at most one unreliable detection result; the pre-annotation complexity of this frame of data is defined as the average intersection over union value of all successfully matched pairs.

[0070] Step 105, correct the pre-annotation result of the unreliable modality according to the intersection over union to obtain the final annotation result of the unreliable modality.

[0071] In one possible implementation, the pre-annotation result of the unreliable modality is corrected according to the intersection over union, including:

[0072] If the intersection over union is greater than a first threshold, output the pre-annotation result of the unreliable modality as the final annotation result; if the intersection over union is less than the first threshold and greater than a second threshold, determine the weighted annotation result of the unreliable modality based on the projection annotation result of the unreliable modality and the pre-annotation result of the unreliable modality, and output the weighted annotation result of the unreliable modality as the final annotation result; if the intersection over union is less than the second threshold, call the target detection large model for secondary annotation, and output the secondary annotation result of the unreliable modality as the final annotation result.

[0073] Specifically, the first threshold can be set to 0.7, and the second threshold can be set to 0.3.

[0074] As shown in Figure 3 According to the intersection over union value output by the target association module, a hierarchical processing strategy is adopted: when the intersection over union is greater than 0.7, it is considered that the detection results of the two modalities are highly consistent, and the pre-annotation result of the unreliable modality is directly used; when the intersection over union is between 0.3 and 0.7, a frame fusion algorithm based on reliability weighting is used to generate the final annotation result; when the intersection over union is less than 0.3, it is recognized as a significant difference, which needs to be further analyzed and processed.

[0075] In a possible implementation, the weighted labeling result of the non-reliable modality includes: determining a weight coefficient according to the modality reliability, and performing weighted fusion on the bounding box coordinates of the projection labeling result and the pre-labeling result to obtain the weighted labeling result.

[0076] Specifically, for the target pair that the matching is successful and the intersection over union is between 0.3 and 0.7, a reliability weighted bounding box fusion algorithm is used to generate a final labeling result, and a specific fusion formula is as follows:

[0077]

[0078] wherein, represents the final bounding box coordinates after fusion, represents the projection bounding box coordinates of the reliable modality, represents the pre-labeling bounding box coordinates of the non-reliable modality.

[0079] The calculation formula of the weight coefficient a is as follows: wherein represents the reliability score of the reliable modality under the current environmental condition, represents the reliability score of the non-reliable modality under the current environmental condition.

[0080] In a possible implementation, the secondary labeling performed by calling the target detection large model includes: expanding the detection region with the bounding box of the projection labeling result or the pre-labeling result as the center, and re-performing target detection after adjusting the detection threshold.

[0081] Specifically, for the case of significant difference with the intersection over union less than 0.3, the following two cases are processed:

[0082] When the reliable modality detects and the non-reliable modality does not detect, a target detection region with a scale expanded by 2 times is constructed based on the projection region center of the reliable modality on the non-reliable modality. The detection threshold is reduced in the region to re-perform the large model pre-labeling. If an effective detection result is obtained, it is confirmed that the non-reliable modality is missed detection, and the projection result is used as the final labeling. If no effective detection result is obtained, it is confirmed that the inconsistency is caused by occlusion, and the target is excluded in the non-reliable modality.

[0083] When the non-reliable modality detects and the reliable modality does not detect, a target detection region with a scale expanded by 2 times is constructed based on the detection region center of the non-reliable modality. The detection threshold is improved in the region to re-perform the large model pre-labeling. If an effective detection result is still obtained, it is confirmed that the inconsistency is caused by occlusion, and the non-reliable modality detection result is used as the final labeling. If no effective detection result is obtained, it is confirmed that the non-reliable modality is false detection, and the target is excluded in the final result.

[0084] In summary, in the embodiment of the present application, the target detection large model is used to uniformly pre-annotate multi-modal data including infrared images, effectively solving the problem of missing infrared data annotation in the prior art, and being able to process the characteristics unique to infrared images such as low contrast and texture blur. The environmental parameter analysis module is used to evaluate the reliability of each sensor in real time, and the optimal mode is dynamically selected based on environmental parameters such as light intensity and visibility, overcoming the performance limitations of a single mode in specific environmental conditions. The spatial projection module is used to realize cross-modal annotation mapping, and the intersection-over-union analysis and multi-level decision mechanism of the target association module are used to effectively eliminate the temporal and spatial synchronization errors and calibration parameter deviations between multi-modal sensors. Through weighted fusion and secondary annotation mechanism, the annotation quality is improved while ensuring the annotation efficiency, and when there is a large deviation in the pre-annotation result, secondary annotation is automatically started to ensure the accuracy and reliability of the final annotation result. Through the synergistic effect of environmental perception mode selection and multi-level annotation correction, the present application effectively solves the technical problems of poor environmental adaptability and insufficient multi-modal data fusion accuracy in the prior art.

[0085] In a possible implementation, the modified multi-modal annotation result is stratified sampled according to a pre-defined annotation difficulty grading standard. For annotation data divided into a high difficulty level, an artificial verification and correction process is started to obtain verified annotation results.

[0086] The verification mechanism focuses on reviewing difficult cases and boundary conditions that may exist in the automatic annotation process through human-computer collaboration, effectively ensuring the accuracy and consistency of the annotation results, and optimizing the allocation of artificial review resources.

[0087] The artificial verification link, as an effective supplement to the automatic pre-annotation of the large model, can specifically find and correct possible detail errors in high difficulty samples, especially complex scenes that may be missed or misjudged in the pre-annotation result correction step. Through this difficulty stratified quality verification strategy, the system realizes the best balance between annotation cost and benefit while ensuring annotation quality.

[0088] In a possible implementation, the verified annotation results are stored in a special database and standardized to be compatible with different data interface requirements.

[0089] The database provides data support for subsequent various application scenarios, including but not limited to training of automatic driving perception models, development of target identification and avoidance algorithms for decision systems, etc.

[0090] In the standardization process, the verified annotation results are converted into the commonly used data formats in the industry. For example, the three-dimensional annotation results of the lidar point cloud are converted into the KITTI standard format, and the annotation results of the image data are converted into the commonly used two-dimensional target detection format such as Pascal VOC or COCO according to the application requirements. This standardization process ensures the good compatibility of the generated data with the mainstream algorithm framework and training process, greatly improves the practical value and sharing convenience of the data.

[0091] In yet another embodiment of the present application, an electronic device is provided, which includes a processor and a memory, and the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the multi-modal data annotation method proposed in the embodiments of the present application.

[0092] In yet another embodiment of the present application, a computer-readable storage medium is provided, which stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the multi-modal data annotation method proposed in the embodiments of the present application.

[0093] The above mainly introduces the scheme provided by the embodiments of the present application from the perspective of the device. It can be understood that the device includes the corresponding hardware structure and / or software module for executing each function in order to realize the above functions. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0094] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for labeling multi-modal data, characterized in that, The method comprises: synchronously collecting multi-modal data and obtaining environmental parameters; the multi-modal data comprises laser radar point cloud data, visible light camera images, and infrared camera images; the environmental parameters comprise light intensity, visibility, precipitation intensity, environmental temperature, and relative humidity; based on the environmental parameters, obtaining modal reliabilities, determining a modal with the highest modal reliability as a reliable modal, and determining other modals as non-reliable modals; performing pre-labeling on the multi-modal data based on a target detection large model to obtain pre-labeling results of the multi-modal data; the pre-labeling results of the multi-modal data comprise pre-labeling results of the reliable modal and pre-labeling results of the non-reliable modals; the pre-labeling results of the reliable modal are final labeling results of the reliable modal; transforming the pre-labeling results of the reliable modal to obtain projection results of the pre-labeling results of the reliable modal on the non-reliable modals; inputting the projection results on the non-reliable modals and the pre-labeling results of the non-reliable modals into a target association module to output intersection over union (IoU); correcting the pre-labeling results of the non-reliable modals according to the IoU to obtain final labeling results of the non-reliable modals. 2.The method of claim 1, wherein, The modal reliabilities comprise laser radar reliability, visible light camera reliability, and infrared camera reliability; the laser radar reliability is determined based on visibility, precipitation intensity, environmental temperature, and relative humidity; the visible light camera reliability is determined based on light intensity, visibility, precipitation intensity, and relative humidity; and the infrared camera reliability is determined based on precipitation intensity, environmental temperature, and relative humidity. 3.The method of claim 1, wherein, The correcting the pre-labeling results of the non-reliable modals according to the IoU comprises: if the IoU is greater than a first threshold, outputting the pre-labeling results of the non-reliable modals as the final labeling results; if the IoU is less than the first threshold and greater than a second threshold, determining weighted labeling results of the non-reliable modals based on the projection labeling results of the non-reliable modals and the pre-labeling results of the non-reliable modals, and outputting the weighted labeling results of the non-reliable modals as the final labeling results; and if the IoU is less than the second threshold, calling the target detection large model to perform secondary labeling, outputting secondary labeling results of the non-reliable modals, and outputting the secondary labeling results of the non-reliable modals as the final labeling results. 4.The method of claim 1, wherein, The transforming the pre-labeling results of the reliable modal comprises: when the pre-labeling results of the reliable modal are point cloud data labeling, mapping three-dimensional point cloud targets to a coordinate system of the non-reliable modals through projection transformation; and when the pre-labeling results of the reliable modal are image data labeling, converting two-dimensional image labeling to three-dimensional space information by using inverse projection technology. 5.The method of claim 3, wherein, The weighted labeling results of the non-reliable modals comprise: determining a weight coefficient according to the modal reliabilities, and performing weighted fusion on boundary box coordinates of the projection labeling results and the pre-labeling results to obtain the weighted labeling results. 6.The method of claim 3, wherein, The calling the target detection large model to perform secondary labeling comprises: enlarging a detection region with the boundary box of the projection labeling results or the pre-labeling results as the center, adjusting a detection threshold, and then re-performing target detection. 7.The method of claim 1, wherein, The method further comprises: According to a predefined labeling difficulty grading standard, the obtained final labeling results of the non-reliable modal are stratified sampled, and for the labeling data divided into a high difficulty level, an artificial verification and correction process is started to obtain verified labeling results. 8.The method of claim 7, wherein, The method further comprises: The verified labeling results are stored in a special database and standardized to be compatible with different data interface requirements.

9. An electronic device, comprising: The electronic device comprises a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or at least one program is loaded and executed by the processor to realize the labeling method of the multi-modal data according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program, the at least one instruction or at least one program is loaded and executed by the processor to realize the labeling method of the multi-modal data according to any one of claims 1-8.

Images