Automatic driving scene labeling method and system for multi-modal data fusion

Abstract

The scheme belongs to the field of automatic driving scene labeling, and specifically relates to an automatic driving scene labeling method and system for multi-modal data fusion. The automatic driving scene labeling method for multi-modal data fusion comprises the following steps: S10: deploying a camera and a laser radar at the same collection position to collect multi-modal data at the same frequency, wherein the multi-modal data comprises two-dimensional image data and three-dimensional point cloud data, the multi-modal data is labeled separately to obtain single-modal labeling data, the single-modal labeling data comprises a target label and a key point, the labeling data of the two-dimensional image data further comprises a bounding box position, and the labeling data of the three-dimensional point cloud data further comprises a space bounding box position. Through automatic alignment and targeted manual correction, the scheme reduces redundant operations, improves labeling efficiency, solves the problem of low labeling efficiency in the existing automatic driving scene labeling method in multi-modal data fusion, and further enhances accuracy and robustness.

Description

TECHNICAL FIELD

[0001] The present solution belongs to the field of automatic driving scene annotation, and specifically relates to an automatic driving scene annotation method and system for multi-modal data fusion. BACKGROUND

[0002] In the automatic driving technology, multi-modal data fusion is a process of collecting and integrating environmental information through various sensors such as lidar and camera. Lidar provides three-dimensional point cloud data to present the spatial structure, and camera captures visual images to provide semantic information. The fusion of the two can improve the comprehensive cognition of the environment by the automatic driving system. Scene annotation, as a key link, needs to add semantic labels to multi-modal data to clearly define the category, location, and cross-modal correlation of the target.

[0003] However, in the driving scene annotation, the fusion of multi-modal data faces serious alignment problems. There is a natural difference between the three-dimensional coordinate system of lidar and the two-dimensional pixel coordinate system of camera, and the sensor external parameters drift over time (minor changes in installation angle, vibration displacement, etc.), resulting in the vehicle outline in the point cloud often deviating from the true pixel position in the image after projection, which requires manual correction of the external parameter matrix. At the same time, lidar can penetrate through obstacles to identify the three-dimensional shell of a vehicle, but cannot determine the color or traffic light status; images can distinguish the color of traffic lights, but it is difficult to perceive occluded objects. This complementary information with missing information makes it difficult for automatic algorithms to establish reliable cross-modal correlation, and still requires manual confirmation of the identity of "point cloud column" and "image lamp pole".

[0004] Currently, the automatic driving scene annotation method cannot solve the above problems in multi-source data fusion. For example, Chinese patent CN118313485A focuses on single-modal automatic annotation, and Chinese patent CN118628999A still relies on manual high-frequency checking for cross-modal correlation. The annotator needs to view the point cloud and image simultaneously, manually calibrate the parameters, and match the semantics frame by frame, which is several times the amount of operation of single-modal annotation. Moreover, subjective judgment differences reduce the consistency of annotation, resulting in low annotation efficiency in multi-source data fusion. SUMMARY

[0005] The purpose of the present solution is to provide an automatic driving scene annotation method and system for multi-modal data fusion to solve the problem of low annotation efficiency in the existing automatic driving scene annotation method in multi-modal data fusion.

[0006] To achieve the above purpose, the present solution provides an automatic driving scene annotation method for multi-modal data fusion, comprising the following steps:

[0007] S10: deploying a camera and a lidar to collect multi-modal data at the same collection position, the multi-modal data including two-dimensional image data and three-dimensional point cloud data, the multi-modal data being respectively labeled to obtain single-modal labeled data, the single-modal labeled data including a target label and a key point, the labeled data of the two-dimensional image data further including a bounding box position, and the labeled data of the three-dimensional point cloud data further including a space bounding box position;

[0008] S20: in the key point of the labeled data of the three-dimensional point cloud data, an alignment point is selected according to the relative position between the key point and the collection position, the alignment point of the three-dimensional point cloud data is projected to the two-dimensional image data according to the camera parameters, and the key point in the two-dimensional image data is matched as the alignment point according to the coordinates of the projection result in the two-dimensional image data; a space conversion matrix is generated based on the mapping relationship of the multi-modal data alignment point, the geometric alignment of the key point under the multi-modal data is completed, and the single-modal labeled data is associated according to the geometric alignment result;

[0009] S30: consistency verification is performed on the association result, and if:

[0010] S30a: if the target labels of the multi-modal data are different or the key point coordinate difference exceeds a preset difference threshold, the multi-modal data is sent to an artificial terminal; a correction result is received, and the corresponding single-modal labeled data is modified, and the step S30 is returned;

[0011] S30b: if the target labels of the multi-modal data are consistent and the key point coordinate difference is within the difference threshold, the artificial terminal is sent the superposition effect of the target labels, the key points and the bounding box or the space bounding box of the same multi-modal data, and a verification result is received; if the verification result is correct, the step S40 is executed, and if the verification result is not correct, a correction result is received, and the corresponding single-modal labeled data is modified, and the step S30 is returned;

[0012] S40: the multi-modal labeled data of the target object is fused according to the association result.

[0013] The principle and technical effect of the scheme are that: first, the scheme obtains the content that needs to be manually and accurately accessed through automatic preprocessing, and significantly reduces the redundant manual operation and significantly improves the labeling efficiency through the mode of manual and accurate intervention. In the data alignment stage, the relative positions of three-dimensional key points (i.e. key points in three-dimensional point cloud image data) and collection positions are used to filter long-distance high-precision points, and a spatial conversion matrix is used to automatically complete batch point alignment, thereby saving the mechanical work of manual point-by-point comparison and manual coordinate calibration in traditional methods, and reducing most of the comparison operations in complex scenes containing hundreds of multi-modal data key points. In the consistency verification stage, the scheme first automatically judges whether the target label conflict and key point coordinate difference of multi-modal data (i.e. two-dimensional image data and three-dimensional point cloud data) exceed the threshold value, and triggers manual intervention when the threshold value is exceeded. When manual correction is performed, the single data is adjusted first (such as correcting the three-dimensional point cloud data first and then comparing the image data and the three-dimensional point cloud data), and the modified content still cannot achieve the ideal effect through automatic processing in the common scene of adapting to only view the single data modification, avoiding meaningless bidirectional modification, and reducing repeated operations compared with the traditional manual full-frame comparison mode. At the same time, the association result directly provides a labeling basis containing two target labels, two key points and a frame, and the manual operation does not need to label from scratch or manually find the cross-modal corresponding points, but only needs to confirm or correct the result, thereby further saving the steps of repeatedly labeling all key points and switching cross-modal data to associate points. This design greatly reduces the manual labeling time of a single scene, efficiently adapts to the large-scale scene data labeling needs of autonomous driving, and solves the core problem of high proportion of manual operation and low efficiency in traditional multi-modal labeling.

[0014] In the data acquisition and single-modal labeling link, the scheme deploys sensors at the same position and frequency and synchronously collects data, ensures the initial correlation of two-dimensional image data and three-dimensional point cloud data in time and space dimensions, and provides original time and space anchor points for subsequent alignment. At the same time, the standardized dimensions of the labeling target label, frame and key point are unified, which not only provides basic data for multi-modal fusion, but also forms a unified and universal labeling language across scenes and sensors, facilitates the reuse and sharing of labeling data sets in the industry, reduces the adaptation cost of different algorithms to labeling formats, and solves the data island problem caused by format chaos in traditional labeling.

[0015] In the relative position-based geometric alignment link, in addition to using long-distance points to improve accuracy, such a method can adapt to the scene dynamically: in a close-range complex scene (such as dense traffic), through camera parameters, including intrinsic and extrinsic parameters, a small number of reliable long-distance points are automatically focused to reduce errors; in a long-distance open scene (such as a highway), the high accuracy of the coordinates of the large-range three-dimensional key points is fully utilized to realize scene adaptive alignment and reduce the operation of manually adjusting algorithm parameters; at the same time, the batch alignment of the space conversion matrix makes the alignment logic traceable, which is convenient for later backtracking and error source tracking, and breaks through the unexplainable limitations of traditional black-box alignment algorithms.

[0016] In the data fusion link, by integrating the texture details (such as color-based target appearance features) of two-dimensional image data and the spatial positions (such as target depth and volume features) of three-dimensional point clouds, not only more comprehensive labeling information is generated, but also cross-modal complementary fault tolerance in extreme environments is realized. For example, in a rainstorm, when two-dimensional image data is blurred by rain and fog, the spatial bounding box of three-dimensional point clouds can provide accurate position reference; in a strong light environment, when three-dimensional point cloud data is offset due to reflection interference, the texture features of two-dimensional images can assist in correcting the label, so that the labeling data remains reliable in complex working conditions, providing key support for the training of automatic driving algorithms in extreme scenes.

[0017] In summary, the present scheme reduces redundant operations, improves labeling efficiency, and solves the problem of low labeling efficiency in multi-modal data fusion of existing automatic driving scene labeling methods, and also enhances accuracy and robustness.

[0018] Further, the alignment points in the three-dimensional point cloud data labeling data are taken as three-dimensional alignment points, and the alignment points in the two-dimensional image data labeling data are taken as two-dimensional alignment points; the key points in the three-dimensional point cloud data labeling data are taken as three-dimensional key points, and the key points in the two-dimensional image data labeling data are taken as two-dimensional key points; in the step S20, when the three-dimensional alignment points are screened, the relative positions of the three-dimensional key points and the collection positions are used to determine a preliminary candidate range, and then the two-dimensional key point features corresponding to the preliminary candidate range are adaptively adjusted, so that the three-dimensional alignment points finally selected form an adaptive balance between spatial distance and image feature quality.

[0019] The scheme collects two-dimensional image and three-dimensional point cloud data at the same position and frequency, and labels target labels, bounding boxes (space bounding boxes) and key points respectively, which lays a foundation for multi-modal fusion. In the alignment process, the relative position between the three-dimensional key points and the collection position is used to filter the alignment points. The farther the relative position is, the more accurate the point collected by the image data is, and the smaller the error is when aligning, thereby significantly improving the accuracy of alignment. Further adaptive adjustment is combined with two-dimensional key point features: when filtering three-dimensional alignment points, first determine the preliminary candidate range that matches the spatial distance characteristics according to the depth distribution of the point cloud data to ensure that the initial candidate points conform to the reliable distribution of the three-dimensional space; then, through two-dimensional image feature checking, remove distant but blurred points and supplement close but clear points, so that the final selected three-dimensional alignment points form an adaptive balance between spatial distance and image feature quality. This way not only avoids the introduction of low-quality points caused by relying solely on distance, but also enhances the stability of alignment through adaptive scene adjustment, especially in different scenarios such as urban complex sections (distant points are easily obscured and blurred) and highways (distant points are clearer). Thus, the alignment accuracy is improved, the manual operation is reduced, and the labeling efficiency is improved.

[0020] Further, when sending the associated two-dimensional image data and three-dimensional point cloud data to the artificial terminal for consistency verification, the associated results corresponding to the three-dimensional alignment points with a relatively long distance from the collection position are preferentially provided; then, the verification results are collected, the changes of the two-dimensional alignment points corresponding to the three-dimensional alignment points in the correct verification results are extracted as offset features, and a relative position between the three-dimensional alignment points and the collection position and the offset features are used to establish an offset rule model. For two-dimensional image data and three-dimensional point cloud data without obtaining verification results, the relative position between the key points in the three-dimensional point cloud data and the collection position is input into the offset rule model to calculate the offset feature, and the two-dimensional alignment points corresponding to the three-dimensional alignment points are adjusted according to the offset feature.

[0021] By preferentially pushing the correlation results of long-distance three-dimensional alignment points, the scheme takes advantage of the high accuracy of these points themselves, reduces the correction cost of manual verification, focuses the manual effort on high-quality data, and significantly improves the verification efficiency. The offset law model established based on relative position and offset characteristics not only can automatically adjust the alignment points of unverified data to reduce manual intervention, but also can feed back to the early point selection through accumulated rules, such as finding that the offset is frequent in a certain distance interval, reducing the point selection in this interval in the future, forming a closed loop between verification, learning and optimization. At the same time, the unified variables "relative position" and "offset characteristics" not only serve as the input of the offset law model to realize automatic adjustment, but also provide the basis for deviation tracing for manual verification, clarify the reason for deviation, and improve the efficiency of human-machine cooperation. In addition, the self-optimizing ability of the offset law model enables it to dynamically adapt to changes in the scene, such as automatically adjusting the alignment points in heavy rain without the need for manual reconfiguration of rules, significantly improving the robustness and adaptability of the scheme. For example, in the highway scene, the guardrail points at 100-200 meters are preferentially pushed to manual verification, and the offset law model learns that the slight left deviation in this interval is caused by the radar scanning angle. For subsequent unverified guardrail points in the same interval, it is automatically adjusted to the right, and it is found that the 50-80 meter interval has large fluctuations due to large vehicle shielding, so it actively reduces the points in this interval next time, thereby improving the alignment accuracy in complex traffic scenarios, significantly better than traditional static alignment methods.

[0022] Further, when calculating the offset characteristics, the offset law model first divides the distance levels according to the relative distance between the target object and the collection position, and then divides the size categories according to the contour size of the target object in the two-dimensional image within each level. For each combination of distance level and size category, an offset law sub-model is established to learn the offset law of different distance level and size category combinations. For unverified data, determine the distance level and size category to which it belongs, call the corresponding offset law sub-model to calculate the offset characteristics and adjust the two-dimensional alignment points, so that different distances and sizes of target objects can be adapted and corrected.

[0023] The scheme realizes multiple effects by establishing offset rule sub-models according to the distance level and size category of the target object. First, for target objects of different distances and sizes (such as a small-size traffic sign at a long distance, a medium-size car at a medium distance, and a large-size truck at a short distance), the offset rule sub-models can accurately capture the unique offset rules (such as the proportional offset of a long-distance target due to imaging scaling, the angular offset of a medium-distance target due to sensor view angle differences, and the edge offset of a short-distance target due to occlusion), avoiding the generalization error of a single model for complex scenes and significantly improving the alignment accuracy of various targets. Second, the combination of distance level and size category is used as a unified division standard, which not only realizes differentiated processing of the same target at different distances (such as a truck being divided into different combinations due to size changes at 50 meters and 10 meters, and the offset rule sub-models adapting to its offset rules), but also provides a modular basis for model expansion. When new target types (such as special vehicles) are added, only the offset rule sub-models for the corresponding combinations need to be supplemented, without the need to reconstruct the overall offset rule model, reducing the iteration cost of the scheme. More notably, this hierarchical design gives the offset rule model cross-scene adaptive ability. For example, in an urban expressway scene, small-size signs at a long distance are prone to dynamic blur offset due to high speed and short imaging time, and the offset rule sub-model can learn the blur offset rule in this scene. In a congested road section, a large-size bus at a short distance produces random offset due to frequent occlusion, and the corresponding sub-model can quickly adapt to avoid calibration failure caused by scene switching.

[0024] Further, in the data fusion process, natural language records and attribute descriptions of the target object in the collection scene are collected synchronously; the text descriptions of the natural language records and the attribute descriptions are analyzed, and key semantic elements are extracted, including target categories, state attributes, scene features, and spatial relationships; according to the two-dimensional image data and the three-dimensional point cloud data, the target categories and the state attributes are matched and associated with target labels, and the scene features and the spatial relationships are matched and associated with key features, to establish a mapping relationship between the key semantic elements and the single-modal annotation data; the consistency of the text descriptions and the single-modal annotation data in semantic elements is checked, and when the semantic elements conflict with the single-modal annotation data, the text descriptions and the single-modal annotation data are sent to an artificial terminal, and an artificial verification result is received; the artificial verification result is integrated into the single-modal annotation data after geometric alignment through the text descriptions.

[0025] The scheme parses the natural language description into semantic elements, and maps them with two-dimensional and three-dimensional labeling data, so that the labeling results contain not only spatial positions, but also semantic information such as target dynamic characteristics and scene background, significantly improving the richness. For example, the text description of the slow-moving vehicles at the intersection can be complementary to the vehicle position in the image and the speed trend in the point cloud, providing a more realistic basis for the algorithm. The semantic consistency checking mechanism can actively identify conflicts between text and labeling data, such as when the text mentions "no left turn" but the labeling is not associated with the corresponding sign, triggering manual checking to avoid ignoring semantic contradictions and reducing the risk of algorithmic misjudgment. At the same time, by focusing on conflict points, the manual redundant operations are reduced, and the accuracy and efficiency are balanced. The dynamic fusion of text and labeling data forms a closed-loop optimization, and the manual correction results reinforce the semantic mapping logic in reverse, enabling the scheme to gradually master the semantic association rules of complex scenes, reducing the repeated occurrence of similar conflicts, and enhancing the self-adaptive ability to various scenes, providing high-quality labeling input with spatial precision and semantic depth for autonomous driving.

[0026] Further, the artificial terminal is configured with an image acquisition component to capture the operator's facial information in real time and extract visual attention features, and analyze the operator's gaze direction based on the timing of feature changes; in the S30a step, when the artificial terminal receives two-dimensional image data and three-dimensional point cloud data, it displays the two-dimensional image data and the three-dimensional point cloud data in a side-by-side manner; by identifying the operator's gaze direction and the content displayed at different positions of the artificial terminal, the operator's current focus on single-modal data is highlighted; after the operator enters the correction results of the single-modal labeling data on one side, the other side's single-modal labeling data is adjusted based on the offset rule model, and the adjusted content is highlighted in the display area on the other side.

[0027] The scheme captures the operator's facial visual attention features in real time on the same artificial terminal, highlights the single-modal labeling data being gazed at after displaying the two-dimensional image data and the three-dimensional point cloud data side by side, so that the operator can quickly lock the target to be verified (such as one side of the data) without additional operations, reducing the time-consuming mechanical operations and improving the verification smoothness. When the operator completes the entry of the correction results on one side, the aligned labeling on the other side is automatically adjusted based on the correction results, and the correction results are highlighted again to attract the operator's attention, while emphasizing the content related to the correction results, forming a closed loop of gaze focusing and correction linking, which greatly reduces the operation cost of switching between two-modal data and makes the labeling process more coherent.

[0028] In addition, the gaze trajectory can be recorded as an attention heat map, and the inverse optimization algorithm is used in subsequent offset law model training. The offset law model is trained to preferentially learn the areas of human focus in complex scenes (such as rain, snow, backlight, and congested road conditions), and the algorithm's ability to capture key details is enhanced. For example, in a rainy night road scene, when the operator's gaze sweeps across the distant blurred guardrail, the artificial terminal immediately highlights the corresponding point cloud and image area; if the operator adjusts the labeled boundary of the guardrail, the artificial terminal immediately updates and highlights the new alignment range on the other side, and at the same time, the features of this gaze focus area are fed back to the offset law model, so that the automatic labeling of similar scenes in the future can more accurately identify similar targets, further improving the labeling reliability in complex environments.

[0029] Further, after the artificial terminal enters the correction result, the single-mode labeling data before correction and the correction result are recorded to form training samples; the labeling learning model is established using the training samples, and the labeling learning model is used to calculate the correction result as a predicted correction result according to the current single-mode labeling data; when the artificial terminal displays the two-dimensional image data and the three-dimensional point cloud data side by side, the predicted correction result is calculated according to the current single-mode labeling data and the gaze direction, and the predicted correction result is superimposed and displayed in the corresponding single-mode labeling data; then the predicted correction result and the current single-mode labeling data are put into the offset law sub-model to adjust the other single-mode labeling data to obtain a predicted adjustment result, and the predicted adjustment result is superimposed and displayed in the other single-mode labeling data.

[0030] The present scheme forms training samples by recording the labeling data before and after correction and constructs a labeling learning model, which can calculate a predicted correction result based on the current labeling data and the operator's gaze direction, and obtain a predicted adjustment result on the other side through the offset law model and superimpose it for display, which significantly improves the efficiency of artificial labeling: when the operator views the bimodal data side by side, the operator can directly refer to the pre-displayed correction and adjustment results, without the need to start from zero to determine the correction direction, reducing the time spent on thinking and operation; for accurate prediction results, the operator only needs to confirm to complete the labeling, greatly shortening the processing time of a single data; even if there is a deviation in the prediction, the operator can quickly adjust based on the pre-displayed content, reducing the amount of repeated operation, while the model continuously learns the artificial correction law, and the prediction accuracy is continuously improved, further reducing the depth and frequency of human intervention, making the multi-modal data labeling process more efficient and smooth.

[0031] Further, before training the labeling learning model, semantic elements contained in the training samples are extracted, the association between the semantic elements and the current single-modal labeling data and the correction result is analyzed, and the feature difference of the correction result under different semantic elements is determined; the training samples are classified according to the target labels and key points contained in the semantic elements and the correction result, and the classified training samples are sequentially input into the labeling learning model for training, so that the labeling learning model learns the correction rule of a certain type first, and then gradually incorporates other types of training samples, until the labeling learning model can form corresponding prediction logic for different semantic elements and correction types.

[0032] By extracting the semantic elements in the training samples and analyzing the association between the semantic elements and the correction result, the labeling learning model can gradually learn different types of correction rules. Not only can it capture explicit scene differences such as rain and sunny vehicle bounding box adjustment and correction parameter adaptation under different lighting conditions, but it can also mine implicit collaborative relationships between semantic elements, such as recognizing the correction linkage of "pedestrian" and "zebra crossing" in intersection scenes. When the pedestrian bounding box deviates, the zebra crossing labeling range will also show corresponding adjustment rules. This cross-target association learning is difficult to achieve with traditional single-target training models. The way of sequentially inputting training samples by category enables the labeling learning model not only to distinguish between label correction, coordinate adjustment, etc., but also to learn the priority logic of different correction types, such as preferentially adapting the bounding box shrinkage caused by excessive camera exposure in backlight scenes, and then adjusting the point cloud label semantic matching, which far exceeds the simple classification training effect. Moreover, this way enables the labeling learning model to have a rapid learning ability for rare edge cases, such as the confusion of "construction cone" and "guardrail" in sandstorm weather. Even if the sample size is extremely small, it can quickly establish targeted rules through the association classification of semantic elements such as "low visibility" and "road construction" and the correction result; at the same time, it dynamically distinguishes the hierarchical relationship of semantic elements, such as learning the differentiated correction rules of "truck" (both prone to point cloud projection deviation due to size) and "sedan" (prone to label misjudgment due to color recognition dependence on images) under the "vehicle" category. This semantic-driven hierarchical training mechanism not only reduces manual intervention, improves labeling accuracy and efficiency through prediction of correction results, but also enables the model to derive correction logic through existing semantic associations when facing unseen complex scene combinations, greatly improving the robustness and self-adaptation ability of the present scheme in extreme environments and mixed scenes.

[0033] Furthermore, when performing consistency verification on the association results, annotation tasks are generated based on the verification results combined with 3D point cloud data and 2D image data. The priority of the annotation tasks is evaluated based on the relative position between the acquisition location and the target object in the annotation task. Then, the priority of the annotation tasks is adjusted based on the number of times the same target label is repeated in the annotation tasks. When the verification result entered by the manual terminal is correct, the single-modal annotation data corresponding to the annotation task is highlighted in the single-modal annotation data on one side according to the priority of the annotation task.

[0034] This approach enables high-priority tasks to directly address annotation challenges, providing high-value samples for the model, avoiding interference from inefficient data, significantly accelerating model convergence, and improving training efficiency. Furthermore, it creates a closed loop between data, the model, and the task; once the offset pattern model or its sub-model grasps the patterns, it automatically lowers the priority of similar tasks, reducing repetitive manual operations and focusing on biases in new scenarios. Moreover, it prioritizes processing related data such as dynamic obstacles, allowing the offset pattern model or its sub-model to learn the deep logic of how motion states affect annotation bias, achieving cross-dimensional pattern learning. Finally, it shifts the offset pattern model or its sub-model from passive correction to active prediction, reducing manual steps while simultaneously promoting a synergistic improvement in annotation efficiency and the capabilities of the offset pattern model or its sub-model. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the autonomous driving scene annotation method for multimodal data fusion in an embodiment of the present invention. Detailed Implementation

[0036] The following will describe the concept and technical effects of the present invention clearly and completely with reference to embodiments, so as to fully understand the purpose, features and effects of the present invention. Obviously, what is described are only some embodiments of the present invention, not all embodiments. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are all within the scope of protection of the present invention.

[0037] like Figure 1 As shown, the autonomous driving scene annotation method for multimodal data fusion includes the following steps:

[0038] S10: Deploy a camera (usually a vehicle-mounted camera) and a lidar (usually a vehicle-mounted lidar) at the same acquisition location to acquire multimodal data at the same frequency. The multimodal data includes two-dimensional image data and three-dimensional point cloud data. The multimodal data is labeled with single-modal annotations to obtain single-modal labeled data. The single-modal labeled data includes target labels and key points. The labeled data of the two-dimensional image data also includes the bounding box position, and the labeled data of the three-dimensional point cloud data also includes the spatial bounding box position.

[0039] S20: In the key point position of the three-dimensional point cloud data annotation data, the alignment point position is screened according to the relative position between the key point position and the collection position. The alignment point position of the three-dimensional point cloud data is projected to the two-dimensional image data in combination with the camera parameters, and the key point position in the two-dimensional image data is matched as the alignment point position according to the coordinate of the projection result in the two-dimensional image data; a space conversion matrix is generated based on the mapping relationship of the multi-modal data alignment point position, the geometric alignment of the key point position under the multi-modal data is completed, and the single-modal annotation data is associated according to the geometric alignment result;

[0040] S30: The consistency of the association result is checked, and if:

[0041] S30a: If the target label of the multi-modal data is different or the difference between the key point positions exceeds the preset difference threshold (set by the administrator according to the calculation accuracy), the multi-modal data is sent to the artificial terminal; then the correction result is received, and the corresponding single-modal annotation data is modified, and the step S30 is returned;

[0042] S30b: If the target label of the multi-modal data is consistent and the difference between the key point positions is within the difference threshold, the artificial terminal is sent to the target label, the key point position and the superposition effect of the bounding box or the space bounding box containing the same multi-modal data, and then the verification result is received; if the verification result is correct, the step S40 is executed, and if the verification result is not correct, the correction result is received, and the corresponding single-modal annotation data is modified, and the step S30 is returned;

[0043] S40: The multi-modal annotation data of the target object is fused according to the association result.

[0044] Among them, the alignment point position in the three-dimensional point cloud data annotation data is taken as the three-dimensional alignment point position, and the alignment point position in the two-dimensional image data annotation data is taken as the two-dimensional alignment point position.

[0045] In the single-modal annotation data, the contour is annotated according to the pixel color change gradient, and the geometric features of the contour are recognized. Then, the target label corresponding to the geometric features and the contour is recognized by the trained machine learning model. The database stores preset key points corresponding to the target label, and the two-dimensional key points are annotated in the two-dimensional image data according to the preset key points. For three-dimensional point cloud data, the recognized point cloud data is matched with the target model stored in the database, and the target label associated with the target model is obtained as the target label of the three-dimensional point cloud data according to the matching result. The three-dimensional key points of the three-dimensional point cloud data are obtained according to the preset key points of the target model. The target label associated with the local structure, the local structure contour, and the local structure relationship is stored in the database. The selection of the key points is related to the local structure and the local structure contour associated with the target label. For example, for an object with a target label of "car", the target label is associated with a wheel structure, a wheel contour, a head contour, a head structure, a tail structure, a tail contour, a vehicle shape, a vehicle shape contour, and a vehicle size. The preset key points are generally point positions at different structure preset positions, such as the upper edge point position and the lower edge point position on the wheel contour.

[0046] In the three-dimensional key points, the alignment point is selected according to the relative position between the key point and the collection position. The collection position of the three-dimensional point cloud data is taken as the origin, and the origin and the key point are connected by a straight line to form a collection line. The angle between the collection lines is generally preset to a screening angle (set by the administrator according to the calculation accuracy). A point on the outermost side of the contour corresponding to the target label is randomly selected as an initial alignment point. The collection line of the initial alignment point is taken as an initial line, and the key point corresponding to the collection line with an angle not less than the screening angle with the initial line is taken as the alignment point. Based on the existing alignment point, the collection line corresponding to the alignment point on the outermost side in each direction is taken as the initial line, and the alignment point is obtained again according to the initial line. The above operation is repeated until the alignment points are uniformly distributed in the range formed by the key points with a distribution threshold higher than the preset distribution threshold (set by the administrator according to the accuracy).

[0047] The camera parameters include distortion adjustment parameters, focusing parameters, rotation matrices, translation vectors, and the like when the camera of the type shoots an image. The two-dimensional image data and the mark data in the two-dimensional image data are adjusted according to the parameters. The two-dimensional key points are adjusted using the distortion adjustment parameters and the focusing parameters, so as to obtain the actual contour of the object corresponding to the target label. The alignment point is found in the annotation data of the two-dimensional image data according to the relative position of the alignment point in each component, and the corresponding key point is taken as the two-dimensional key point. The mapping relationship is established between them according to the matching relationship, and the space conversion matrix is generated according to the mapping relationship.

[0048] In the step S20, when screening the three-dimensional alignment points, a preliminary candidate range is first determined based on the relative positions of the three-dimensional key points and the collection position, and then the two-dimensional key point features corresponding to the preliminary candidate range are adaptively adjusted, so that the final selected three-dimensional alignment points form an adaptive balance between spatial distance and image feature quality.

[0049] In another specific manner, when screening the three-dimensional alignment points, the depth distribution characteristics of the three-dimensional point cloud data are first analyzed to determine the distribution density and reliability of the key points in the point cloud data in different distance intervals. For example, in a highway scene, the point cloud data may exhibit a distribution characteristic that the middle and long distance points are dense and the short distance points are sparse, and in a city road scene, the short distance points may be dense and contain more occlusion details.

[0050] Based on the above depth distribution characteristics, the key points in the point cloud data are divided into multiple intervals according to the relative distance from the collection position, and a distance interval with high distribution density and small noise interference is selected as a preliminary candidate range, so that the range is adapted to the natural depth distribution of the point cloud data. For example, in a highway scene, the "50-200 meters" interval (different intervals are set by the administrator for different road scenes) is preferentially selected as the preliminary candidate range, and in a city road scene, the "10-80 meters" interval is preferentially selected as the preliminary candidate range.

[0051] Then, the three-dimensional key points in the preliminary candidate range are projected onto the two-dimensional image plane, and the definition characteristics of the corresponding two-dimensional key points are extracted, including contour integrity, edge sharpness and detail recognizability; if the corresponding two-dimensional key points of a three-dimensional point exist serious blur, occlusion or detail loss, the three-dimensional point is excluded from the preliminary candidate range.

[0052] Finally, from the distance interval outside the preliminary candidate range but still having a certain reliability according to the point cloud depth distribution, three-dimensional points with higher definition of the corresponding two-dimensional key points are supplemented to form the final set of three-dimensional alignment points, which not only ensures the adaptability of the point source to the point cloud depth distribution, but also balances the spatial distance and image quality through two-dimensional feature verification, and improves the stability of subsequent geometric alignment.

[0053] When sending associated 2D image data and 3D point cloud data to the human terminal for consistency verification, priority is given to providing the associated results corresponding to 3D alignment points that are far from the acquisition location. Then, the verification results are collected, and the changes in 2D alignment points corresponding to 3D alignment points in the correct verification results are extracted as offset features. An offset rule model is established by combining the relative position of 3D alignment points with the acquisition location and the offset features. For 2D image data and 3D point cloud data for which no verification results have been obtained, the relative position between key points in the 3D point cloud data and the acquisition location is input into the offset rule model to calculate the offset features. The 2D alignment points corresponding to the 3D alignment points are adjusted according to the offset features.

[0054] Specifically, when sending data to the human terminal for consistency verification, the data is sorted from farthest to closest according to the distance between the 3D alignment points and the acquisition location. The association results that are farthest away and have clear features in the 2D image are pushed first, along with the distance information of that point for human reference. After collecting the verification results returned by the human terminal, the offset features of the 2D alignment points relative to the 3D alignment points are extracted. The offset of the 2D alignment points can be expressed as the difference between the coordinates of the 3D alignment points and the coordinates of the 2D alignment points. That is, let the spatial coordinates of the 3D alignment points be... The ideal pixel coordinates projected onto the two-dimensional image plane are: ,in, The horizontal pixel coordinates of this ideal point. These are the vertical pixel coordinates of the ideal point. The actual pixel coordinates of the annotated 2D alignment point are... ,in, The horizontal pixel coordinates of this ideal point. Let be the vertical pixel coordinates of this ideal point. Then the offset vector... It is used to quantify offset features; it distinguishes between systematic offsets and random offsets, the former being determined by calculating the average of all offset vectors within a certain distance interval. get, This represents the average horizontal offset. This represents the average vertical offset. It reflects the overall offset trend within this interval due to sensor characteristics; the latter can be obtained by calculating the deviation between the offset vector and the average value, reflecting the local offset caused by temporary occlusion. Combining the relative distance D between the 3D alignment point and the acquisition position (i.e., the straight-line distance between this point and the acquisition origin in 3D coordinates), offset characteristics are categorized according to distance intervals, and an offset pattern model is constructed. Each distance interval in the offset pattern model... Corresponding to a set of typical offset vectors ; Divide the numerical range of the i-th distance interval (e.g., "near distance interval", "medium distance interval", etc.); for the associated data that has not completed consistency verification, extract the distance between its 3D key points and the acquisition location. The distance between the three-dimensional key point and the origin of the collection position in the to-be-verified associated data is used to match the distance interval in the offset regularity model, and a typical offset vector in the matching corresponding interval of the offset regularity model is input The typical offset vector in the matching corresponding distance interval), if it is determined to be systematic offset, the corrected two-dimensional alignment point coordinate is ; if it is determined to be possible random offset, small fine-tuning is performed (wherein k is an adjustment coefficient less than 1, used for small fine-tuning in the "random offset" scenario, controlling the amplitude of correction, set by the administrator) and marked as suspicious; the corrected data reenters the consistency verification process, and the correction result of this time is supplemented to the offset regularity model with subsequent manual verification feedback, when the offset regularity model finds that the typical offset vector changes due to scene changes (such as weather, light), the average value of the offset vector in this interval is updated to realize model iteration and ensure that the correction logic dynamically adapts to scene changes.

[0055] In a specific implementation scenario of the present scheme, for example, in an urban morning peak automatic driving data collection scenario, the vehicle-mounted camera and laser radar synchronously collect front road data, and need to perform multi-modal labeling on targets such as traffic lights, vehicles, and pedestrians in the road. The present scheme first screens three-dimensional alignment points, and preferentially pushes the associated results of the three-dimensional point of the traffic light 100 meters away (far from the collection position, less disturbed) to the manual terminal. When manually verifying, it is found that the ideal pixel coordinates of the three-dimensional point of the traffic light projected to the two-dimensional image have a slight right deviation from the actual labeled two-dimensional point, and this offset feature is recorded.

[0056] After collecting multiple sets of manual verification results, the offset vector is calculated. Assuming that the three-dimensional coordinates of the traffic light are , represent the coordinates of the three-dimensional space point (usually based on the laser radar coordinate system), used to describe the position of the target (such as a traffic light, a vehicle) in the three-dimensional world. The ideal projection pixel coordinates are , the actual labeled two-dimensional coordinates are , and the offset vector is . By analyzing the offset data of multiple long-distance traffic lights, it is found that there is a systematic right deviation (average offset vector ) in the 100-150 meter interval, which is determined to be a sensor characteristic offset caused by a small deviation of the laser radar and camera external parameters; and the offset vector of the vehicle point in the middle distance (30-50 meters) fluctuates greatly (such as , ), which is determined to be a random offset.​

[0057] Based on these features, the offset rule model is constructed, and the offset rule is stored according to the distance interval. For example, the 100-150 meter interval is right-biased by about 3 pixels by default, and the 30-50 meter interval is preliminarily fine-tuned according to the random offset amplitude. When processing a group of unverified 50-meter vehicle data, the distance between the three-dimensional point position of the vehicle and the collection position is extracted, and the offset rule of the 30-50 meter interval is called by matching the model. The ideal projection pixel coordinates of the vehicle are calculated as , combined with the random offset amplitude predicted by the model, the two-dimensional point position is pre-corrected (from the original label to ), and marked as "suspected offset".

[0058] The corrected data enters consistency verification and is completed after labeling. Assuming that at this time, it suddenly rains, and the medium distance (30-50 meters) vehicle is caused by rain and fog, resulting in blurred two-dimensional image. The subsequent manual verification found that the average value of the offset vector in this interval changed from to , and the offset rule in the model was automatically updated in this interval to ensure that the offset change caused by rain and fog could be adapted when correcting the newly collected medium distance vehicle data after the rain, reducing the workload of repeated manual correction.

[0059] More specifically, when the offset rule model calculates the offset features, it first divides the distance levels according to the relative distance between the target object and the collection position, and then divides the size categories according to the outline size of the target object in the two-dimensional image within each level. For each distance level and size category combination, an offset rule sub-model is established to learn the offset rule of different distance level and size category combinations. For unverified data, determine the distance level and size category to which it belongs, call the corresponding offset rule sub-model to calculate the offset features and adjust the two-dimensional alignment point position, so that different distance and size target objects can be adapted and corrected.

[0060] In one specific embodiment of the present scheme, at an urban intersection, when the present scheme processes three groups of targets, the long-distance small-size traffic light (e.g., distance level "far" and size category "small") is corrected by the corresponding offset law sub-model to reduce the alignment error between the two-dimensional point and the three-dimensional point due to the scaling offset caused by the lens focal length; the medium-distance medium-size taxi (e.g., distance level "medium" and size category "medium") is corrected by the offset law sub-model to improve the matching accuracy due to the angle offset caused by the difference between the laser radar and the camera view angle; the near-distance large-size bus (e.g., distance level "near" and size category "large") is corrected by the offset law sub-model to improve the edge alignment accuracy due to the edge offset caused by the shielding of adjacent vehicles. Compared with the traditional method, this targeted processing improves the overall labeling alignment accuracy of complex intersections, and when a "long-distance large-size billboard" target is added, only the corresponding combined offset law sub-model needs to be supplemented to quickly adapt, which reflects the dual advantages of the present scheme in scalability and accuracy.

[0061] In the data fusion process, the natural language record and the attribute description of the target object in the collection scene are synchronously collected; the text description of the natural language record and the attribute description is analyzed, and key semantic elements including target category, state attribute, scene feature and spatial relationship are extracted; according to the two-dimensional image data and the three-dimensional point cloud data, the target category and the state attribute are matched and associated with the target label, and the scene feature and the spatial relationship are matched and associated with the key feature, to establish a mapping relationship between the key semantic elements and the single-modal labeling data; the consistency of the text description and the single-modal labeling data in semantic elements is checked, when the semantic elements conflict with the single-modal labeling data, the text description and the single-modal labeling data are sent to an artificial terminal, and an artificial checking result is received; the artificial checking result is integrated into the single-modal labeling data after geometric alignment through the text description.

[0062] The S30 step further comprises the following steps:

[0063] A10: When the consistency check is performed, first, whether the difference between the target label and the key point coordinate exceeds a preset difference threshold is automatically detected. If the difference exceeds the threshold, an artificial correction process is triggered (the text description and the single-modal labeling data are sent to an artificial terminal, and an artificial checking result is received); if the difference is within the threshold, the next step of checking is entered. Specifically, the preset difference threshold is set differently according to the target type (such as the traffic light coordinate threshold is stricter than the vehicle), the no-difference data is automatically filtered, and only the items exceeding the threshold are pushed to the artificial terminal.

[0064] A20: For the associated results that the superimposed effect is confirmed to be consistent with the target label and the key point coordinate difference is within the difference threshold, the superimposed effect diagram containing the bounding box and the space bounding box of the same target label and key point is generated and sent to the artificial terminal for confirmation. If the artificial confirmation result is correct, the data fusion step is executed; if the confirmation result needs to be corrected, the single-modal labeling data is updated according to the artificial correction result, and the consistency verification step is returned. Specifically, the generated superimposed effect diagram is embedded with a text semantic label (such as "left turning vehicle") to make the artificial terminal more intuitive to compare the geometric position and semantic consistency.

[0065] A30: Dynamically adjusting the difference threshold According to the historical verification result, the preset threshold of the target label and the key point coordinate difference is dynamically adjusted. For example, if a certain type of target object (such as a vehicle) frequently appears with a difference exceeding the threshold in a specific scene (such as a highway), the threshold for this type of target in this scene is appropriately increased to reduce unnecessary correction work. Specifically, a scene, target, and threshold mapping table is constructed based on historical data, for example, the threshold for pedestrian targets in a rainy scene is automatically relaxed by 10% to reduce invalid corrections caused by sensor noise.

[0066] A40: Automatic correction suggestion generation During the artificial correction process, an automatic correction suggestion is generated based on the existing verification data. For example, if a certain type of target object appears in multiple scenes with similar offset patterns, an automatic correction suggestion is generated for the labeling data of this type of target, and the corrected examples are provided for artificial reference. Specifically, the automatic correction suggestion combines the offset rule sub-model to generate a batch adjustment scheme for similar target offset patterns (such as uniformly moving the vehicle bounding box to the left in consecutive curves).

[0067] A50: The artificial verification result is used as feedback information to optimize the subsequent alignment and verification process. For example, if the artificial frequently corrects the labeling data of a certain type of target object, the system records the characteristics of this type of target and automatically adjusts the alignment strategy in subsequent processing to reduce the occurrence of similar errors. Specifically, the artificial correction result is decomposed into semantic mapping rules (such as "construction area" is often associated with "speed limit 30"), which are fed back to the early text analysis link to improve the accuracy of semantic element extraction. This process not only reduces manual operation through automation filtering, but also reduces semantic conflict rate through closed-loop optimization, balancing efficiency and accuracy.

[0068] The artificial terminal is configured with an image acquisition component to capture the facial information of the operator in real time and extract visual attention features, and analyze the gaze direction of the operator in combination with the feature change time sequence. In the S30a step, when the artificial terminal receives two-dimensional image data and three-dimensional point cloud data, the two-dimensional image data and the three-dimensional point cloud data are displayed in a side-by-side manner. By identifying the gaze direction of the operator in combination with the content displayed at different positions of the artificial terminal, the single-mode data currently focused on by the operator is highlighted. After the operator enters the correction result of the single-mode annotation data on one side, the single-mode annotation data aligned on the other side is adjusted in combination with the offset rule model, and the adjusted content is highlighted in the display area on the other side.

[0069] In one specific embodiment of the present scheme, the artificial terminal is equipped with a binocular infrared camera (frame rate 60 fps) installed at the top center of the screen, with an optical axis at an angle of 15° to the screen plane, for real-time capture of the operator's facial image; at the same time, a miniature ToF depth sensor is built-in for light supplement and elimination of environmental light interference.

[0070] A lightweight GazeNet model (e.g. parameter size <1MB) is integrated in the terminal firmware; assuming that the video stream is forward-inferred at a period of 30 ms, and two-dimensional gaze coordinates (x, y) are output; when the gaze coordinates of three consecutive frames fall within the same data window of 50x50 pixels, a "gaze lock" event is triggered.

[0071] The terminal adopts a left-right column layout. For example, the left side displays 1920x1080 two-dimensional image data, and the right side displays a point cloud overhead view rendered by a depth map of the same resolution. When the gaze lock event is triggered, the bounding box / bounding box within the corresponding window is highlighted in green for 500 ms, and a semi-transparent prompt bar "current focus: vehicle-001" is popped up synchronously.

[0072] After the operator completes the correction using the mouse to drag the bounding box in the left image window, the artificial terminal immediately calls the relative distance-based offset rule sub-model to map the correction amount Δx, Δy to the right point cloud window, and automatically adjusts the bounding box vertex coordinates; assuming that the adjusted bounding box is again highlighted in orange for 300 ms to prompt that the linkage has been completed, the terminal records the gaze coordinates in the background at a granularity of 100 ms, generates a 2D heat map and compresses it into a 64x64 grayscale map; after the task is completed, the heat map and the correction record are packaged as JSON and transmitted back to the cloud training node via HTTPS, for self-supervised fine-tuning of the next round of GazeNet, so that the model can preferentially focus on high-frequency gaze areas in future similar scenarios.

[0073] Specifically, after the artificial terminal enters the correction result, the single-modal annotation data before correction and the correction result are recorded to form a training sample; the training sample is used to establish an annotation learning model, which is used to calculate the correction result as a predicted correction result according to the current single-modal annotation data. When the artificial terminal displays the two-dimensional image data and the three-dimensional point cloud data side by side, the predicted correction result is calculated according to the current single-modal annotation data and the gaze direction, and the predicted correction result is superimposed and displayed in the corresponding single-modal annotation data on one side; then the predicted correction result and the current single-modal annotation data are put into the offset rule sub-model to adjust the single-modal annotation data on the other side to obtain a predicted adjustment result, and the predicted adjustment result is superimposed and displayed in the single-modal annotation data on the other side.

[0074] More specifically, before the annotation learning model is trained, semantic elements contained in the training sample are extracted, the association between the semantic elements and the current single-modal annotation data and the correction result is analyzed, and the feature difference of the correction result under different semantic elements is determined; the training sample is classified according to the target label and the key point contained in the semantic elements and the correction result, and the classified training sample is input into the annotation learning model in turn for training, so that the annotation learning model learns the rules of a certain type of correction first, and then gradually incorporates other types of training samples, until the annotation learning model can form corresponding prediction logic for different semantic elements and correction types.

[0075] Among them, when the consistency check result is checked, the annotation task is generated according to the check result combined with the three-dimensional point cloud data and the two-dimensional image data, the priority of the annotation task is evaluated according to the relative position between the collection position and the target object in the annotation task, and then the priority of the annotation task is adjusted according to the number of repetitions of the same target label in the annotation task; when the artificial terminal enters the correct check result, the single-modal annotation data corresponding to the annotation task is highlighted in the single-modal annotation data on one side according to the priority of the annotation task.

[0076] The present scheme generates an annotation task through consistency checking, and dynamically evaluates and adjusts the priority according to the relative position between the collection position and the target object and the number of repetitions of the target label, which not only realizes the fine management of the annotation task, but also brings deep value beyond the conventional efficiency improvement.

[0077] From the perspective of model training, high-priority tasks (such as annotation conflicts of close-range sudden targets and high-frequency label bias) often contain the most critical "annotation difficulties", and these data sets reflect the inherent bias of sensors in complex scenes (such as dense laser radar point cloud at close range but easy overexposure in images, and clear image semantics at long distance but sparse point cloud). Prioritize checking and correcting such data, which can provide "high-value training samples" for the annotation bias model or bias sub-model, allowing it to focus on core bias patterns that determine annotation accuracy without filtering rules in vast amounts of low-conflict, low-information data. The convergence speed of the bias model or bias sub-model is significantly accelerated, and the prediction accuracy in the same training period is much higher than the traditional way of simply increasing the sample size.

[0078] Secondly, this priority ranking forms a positive closed loop of data, model and task: the data corrected by high-priority tasks feeds back to the model, making the bias model or bias sub-model more accurate in annotation prediction for similar scenes, and then in the next task generation, it can automatically identify and reduce the priority of tasks that have solved the bias. For example, when the bias model or bias sub-model masters the annotation rules in backlight through the high-frequency conflict data of "intersection traffic lights", the priority of traffic light annotation tasks in similar scenes will automatically decrease, and artificial operation is only needed to handle new scene bias such as "tunnel entrance and exit traffic lights" that the bias model or bias sub-model has not seen before, greatly reducing the mechanical operation of repeated correction.

[0079] At the same time, the priority based on relative position (such as dynamic obstacles over static environment) ensures that the corrected data contains rich spatio-temporal correlation features, so that the bias model or bias sub-model not only learns geometric alignment rules, but also masters the deep logic of motion state affecting annotation bias (such as the time difference correction of point cloud lagging behind image frames of high-speed vehicles). This cross-dimensional rule learning cannot be achieved by fixed-class training. Finally, by prioritizing data that is most meaningful for the bias model or bias sub-model, this scheme not only reduces the steps of manual annotation, but also promotes the bias model or bias sub-model from passive correction to active prediction, achieving a double spiral rise in annotation efficiency and model capability.

[0080] The above is only an embodiment of the present application, and well-known specific structures and characteristics in the scheme are not described in detail. It should be noted that for those skilled in the art, without departing from the structure of the present application, a number of modifications and improvements can be made, which should be considered as the protection scope of the present application, and these will not affect the effect and practicality of the present application. The protection scope claimed in this application should be subject to the content of its claims, and the specific implementation mode and the like in the description can be used to explain the content of the claims.

Claims

1. An automatic driving scene labeling method for multi-modal data fusion, characterized in that, The method comprises the following steps: S10: deploying a camera and a laser radar to collect multi-modal data at the same collection position, the multi-modal data comprising two-dimensional image data and three-dimensional point cloud data, the multi-modal data being labeled respectively to obtain single-modal labeled data, the single-modal labeled data comprising a target label and a key point, the labeled data of the two-dimensional image data further comprising a bounding box position, and the labeled data of the three-dimensional point cloud data further comprising a space bounding box position; S20: in the key points of the labeled data of the three-dimensional point cloud data, an alignment point is selected according to the relative position between the key points and the collection position; the key points in the labeled data of the three-dimensional point cloud data are taken as three-dimensional key points, and the key points in the labeled data of the two-dimensional image data are taken as two-dimensional key points; the depth distribution characteristics of the three-dimensional point cloud data are analyzed to determine the distribution density and reliability of the three-dimensional key points in different distance intervals; the three-dimensional key points are divided into a plurality of distance intervals according to the relative distance from the collection position, and a distance interval with high distribution density and small noise interference is selected as a preliminary candidate range; then, based on the contour integrity, edge sharpness and clarity features of the two-dimensional key points corresponding to the three-dimensional key points, the preliminary candidate range of the alignment point in the three-dimensional point cloud data is determined; the alignment point of the three-dimensional point cloud data is projected into the two-dimensional image data according to the camera parameters, and the key points in the two-dimensional image data are matched as the alignment points according to the coordinates of the projection result in the two-dimensional image data; a space conversion matrix is generated based on the mapping relationship of the multi-modal data alignment points, the geometric alignment of the key points in the multi-modal data is completed, and the single-modal labeled data is associated according to the geometric alignment result; S30: consistency checking is performed on the association result, if: S30a: if the target labels of the multi-modal data are different or the key point coordinates differ by more than a preset difference threshold, the multi-modal data is sent to an artificial terminal; a correction result input by the artificial terminal is received, and the corresponding single-modal labeled data is modified, and the step S30 is returned; S30b: if the target labels of the multi-modal data are consistent and the key point coordinates differ by less than the difference threshold, the artificial terminal is sent the superposition effect of the target labels, the key points and the bounding box or the space bounding box of the same multi-modal data, and a checking result is received; if the checking result is correct, the step S40 is executed, and if the checking result is not correct, a correction result input by the artificial terminal is received, the corresponding single-modal labeled data is modified, and the step S30 is returned; S40: the multi-modal labeled data of the target object is fused according to the association result; when the associated two-dimensional image data and three-dimensional point cloud data are sent to the artificial terminal for consistency checking, the association result corresponding to the three-dimensional alignment point far from the collection position is preferentially provided. The offset feature is extracted from the change of the two-dimensional alignment point corresponding to the three-dimensional alignment point in the correct verification result, and a offset rule model is established based on the relative position of the three-dimensional alignment point and the collection position and the offset feature. For the two-dimensional image data and the three-dimensional point cloud data without obtaining the verification result, the relative position between the key point in the three-dimensional point cloud data and the collection position is input into the offset rule model to calculate the offset feature, and the two-dimensional alignment point corresponding to the three-dimensional alignment point is adjusted according to the offset feature.

2. The automatic driving scene labeling method for multi-modal data fusion according to claim 1, characterized in that: In the S20 step, when screening the three-dimensional alignment point, the preliminary candidate range is first determined based on the relative position of the three-dimensional key point and the collection position, and then the two-dimensional key point features corresponding to the preliminary candidate range are adaptively adjusted to form an adaptive balance between the spatial distance and the image feature quality of the finally selected three-dimensional alignment point.

3. The automatic driving scene labeling method for multi-modal data fusion according to claim 2, characterized in that: When the offset rule model calculates the offset feature, the distance level is first divided according to the relative distance between the target object and the collection position, and then the size category is divided according to the contour size of the target object in the two-dimensional image in each level; an offset rule sub-model is established for each distance level and size category combination, and the offset rules of different distance level and size category combinations are learned respectively; for unverified data, the distance level and size category to which it belongs are determined, and the corresponding offset rule sub-model is called to calculate the offset feature and adjust the two-dimensional alignment point, so that target objects of different distances and sizes can be adaptively corrected. 4.The automatic driving scene labeling method for multi-modal data fusion of claim 1, wherein: In the data fusion process, the natural language record of the target object collection scene and the attribute description of the target object are collected synchronously; the text description of the natural language record and the attribute description is analyzed, and key semantic elements including target category, state attribute, scene feature and spatial relationship are extracted; according to the two-dimensional image data and the three-dimensional point cloud data, the target category and the state attribute are matched and associated with the target label, and the scene feature and the spatial relationship are matched and associated with the key feature, to establish the mapping relationship between the key semantic elements and the single-modal annotation data; the consistency of the text description and the single-modal annotation data in semantic elements is checked, and when the semantic elements conflict with the single-modal annotation data, the text description and the single-modal annotation data are sent to the artificial terminal, and the artificial verification result is received; the artificial verification result is integrated into the single-modal annotation data after geometric alignment through the text description.

5. The automatic driving scene labeling method for multi-modal data fusion according to claim 3 or 4, characterized in that: The image acquisition component of the artificial terminal is configured to capture the face information of the operator in real time and extract the visual attention features, and analyze the gaze direction of the operator in combination with the feature change time sequence; In the S30a step, when the artificial terminal receives the two-dimensional image data and the three-dimensional point cloud data, the two-dimensional image data and the three-dimensional point cloud data are displayed in a side-by-side manner; By identifying the gaze direction of the operator and combining the content displayed at different positions of the artificial terminal, the single-modal data currently focused by the operator is highlighted; After the operator enters the correction result of the single-modal annotation data on one side, the single-modal annotation data on the other side is adjusted in combination with the offset rule model, and the adjusted content is highlighted in the display area on the other side.

6. The automatic driving scene labeling method for multi-modal data fusion according to claim 5, characterized in that: After the artificial terminal enters the correction result, the single-modal annotation data before correction and the correction result are recorded to form a training sample; The training sample is used to establish a labeling learning model, which is used to calculate a correction result as a predicted correction result according to current single-modal labeling data; When the artificial terminal displays the two-dimensional image data and the three-dimensional point cloud data side by side, the predicted correction result is calculated according to the current single-modal labeling data and the gaze direction, and the predicted correction result is superimposed and displayed in the corresponding single-modal labeling data on one side; then the predicted correction result and the current single-modal labeling data are put into the offset rule sub-model to adjust the single-modal labeling data on the other side to obtain a predicted adjustment result, and the predicted adjustment result is superimposed and displayed in the single-modal labeling data on the other side.

7. The automatic driving scene labeling method for multi-modal data fusion according to claim 6, characterized in that: Before training the labeling learning model, semantic elements contained in the training sample are extracted, the correlation between the semantic elements, the current single-modal labeling data and the correction result is analyzed, and the feature difference of the correction result under different semantic elements is determined; the training sample is classified according to the target label and the key point contained in the semantic elements and the correction result, and the classified training sample is input into the labeling learning model in sequence according to the category for training, so that the labeling learning model learns the rules of a certain type of correction first, and then gradually incorporates other types of training samples, until the labeling learning model can form corresponding prediction logic for different semantic elements and correction types. 8.The automatic driving scene labeling method for multi-modal data fusion of claim 7, wherein: When the consistency of the correlation result is checked, the labeling task is generated according to the checking result in combination with the three-dimensional point cloud data and the two-dimensional image data, the priority of the labeling task is evaluated according to the relative position between the collection position and the target object in the labeling task, and the priority of the labeling task is adjusted according to the number of repetitions of the same target label in the labeling task; When the checking result input by the artificial terminal is correct, the single-modal labeling data corresponding to the labeling task is highlighted in the single-modal labeling data on one side according to the priority of the labeling task.

9. An autonomous driving scene annotation system for multi-modal data fusion, characterized in that, The automatic driving scene labeling method for multi-modal data fusion of any one of claims 1-8 is used.

Images