Map element annotation method, device, electronic device and storage medium
By acquiring and processing the video and point cloud data of the vehicle in the target driving scenario, combining multimodal fusion technology, the pre-trained map element recognition model is used to automatically mark map elements, which solves the inefficiency and accuracy of marking in the existing technology. The problem of low high achieves more efficient and accurate labeling results.
Patent Information
- Application Number
- CN202411063270.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-08-05
AI Technical Summary
In the prior art, map element labeling is inefficient and has low accuracy, mainly due to the efficiency and accuracy problems caused by manual labeling.
By obtaining the video data and point cloud data of the vehicle in the target driving scenario, dynamic target removal and static target coordinate transformation are performed, multi-modal fusion is performed with video data, input into the pre-trained map element recognition model for identification and annotation, and use The recognition results are automatically marked with map features.
Improve the efficiency and accuracy of map feature labeling, reduce the dependence of manual labeling, and enhance the robustness of the identification model and the accuracy of labeling.
Smart Images

Figure CN118587710B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data annotation technology, and in particular to a map element annotation method, device, electronic device and storage medium. Background Art
[0002] Map feature annotation refers to marking various geographic information on a map, such as lane lines, curbs, and zebra crossings. This is crucial because when a vehicle is trained using an autonomous driving model, it can analyze vehicle behavior and predict vehicle trajectories based on the annotated map features.
[0003] However, in the prior art, map elements are usually annotated manually, resulting in low annotation efficiency and low accuracy. Therefore, how to improve the efficiency and accuracy of map element annotation has become a technical problem that needs to be solved urgently. Summary of the Invention
[0004] The present application provides a map element annotation method, device, electronic device and storage medium to solve the problem that map elements are usually annotated manually in the prior art, which has low annotation efficiency and low accuracy.
[0005] In a first aspect, the present application provides a method for annotating map elements, the method comprising:
[0006] Acquire video data and point cloud data collected by the vehicle in a target driving scene, and divide the video data and the point cloud data according to a preset collection time or a preset collection distance to obtain multiple video data subsets and multiple point cloud data subsets;
[0007] Identifying dynamic targets and static targets in each of the plurality of point cloud data subsets, removing point clouds corresponding to the dynamic targets, and performing coordinate transformation on point clouds corresponding to the static targets to obtain a static map corresponding to each point cloud data subset;
[0008] Performing multimodal fusion on the static map and the video data to obtain a bird's-eye view corresponding to each point cloud data subset, and inputting the bird's-eye view into a pre-trained map element recognition model to perform map element recognition and obtain a recognition result;
[0009] The recognition results are used to mark the static map corresponding to each point cloud data subset to obtain the marking results corresponding to each point cloud data subset.
[0010] Optionally, the map element recognition model includes a feature extractor, a regressor and a classifier;
[0011] Inputting the bird's-eye view into a pre-trained map element recognition model to perform map element recognition and obtain a recognition result includes:
[0012] Inputting the bird's-eye view into the feature extractor for feature extraction to obtain feature information corresponding to each map element in the target driving scene;
[0013] Inputting the feature information corresponding to each map element into the regressor for regression processing to obtain the geometric attributes corresponding to each map element, and inputting the feature information corresponding to each map element into the classifier for classification processing to obtain the type corresponding to each map element;
[0014] The geometric attributes corresponding to each map element and the type corresponding to each map element are determined as the recognition result.
[0015] Optionally, the step of inputting the bird's-eye view into the feature extractor for feature extraction to obtain feature information corresponding to each map element includes:
[0016] Adjust the attention weight corresponding to each map element;
[0017] The bird's-eye view is input into the feature extractor, and feature extraction is performed using the adjusted attention weights corresponding to the map elements to obtain feature information corresponding to the map elements.
[0018] Optionally, before inputting the bird's-eye view into a pre-trained map element recognition model to perform map element recognition and obtain a recognition result, the method further includes:
[0019] Iteratively training a model to be trained using a first prediction branch and a second prediction branch, respectively, wherein the data input to the model to be trained by the first prediction branch is a bird's-eye view sample, and the data input to the model to be trained by the second prediction branch is a positive sample and a plurality of negative samples paired with the positive sample, and the model to be trained includes a feature extractor, a regressor, and a classifier;
[0020] A preset loss function is used to calculate the loss value of the model to be trained, and when the loss value is lower than a preset threshold, the iterative training of the model to be trained is stopped to obtain the map element recognition model, wherein the loss value is the sum of the loss value of the first prediction branch and the loss value of the second prediction branch.
[0021] Optionally, performing multimodal fusion on the static map and the video data to obtain a bird's-eye view corresponding to each point cloud data subset includes:
[0022] generating dense point cloud data based on the static map, and generating a texture map based on the video data;
[0023] spatially aligning the dense point cloud data and the texture map;
[0024] The spatially aligned dense point cloud data and the texture map are multimodally fused to obtain the bird's-eye view.
[0025] Optionally, identifying a dynamic target and a static target in each of the plurality of point cloud data subsets includes:
[0026] Acquire target information in each frame of point cloud images in a target point cloud data subset, wherein the target point cloud data subset is any point cloud data subset among the multiple point cloud data subsets, and the target point cloud data subset includes multiple frames of point cloud images;
[0027] Generate a target list corresponding to each frame of the point cloud image according to the target information, wherein the target list is used to record the target type, position, size and motion state contained in each frame of the point cloud image;
[0028] Based on the target list corresponding to each frame of the point cloud image, dynamic targets and static targets in the target point cloud data subset are identified.
[0029] Optionally, the step of labeling a static map corresponding to each point cloud data subset using the recognition result to obtain a labeling result corresponding to each point cloud data subset includes:
[0030] Converting the recognition result into a data structure required by a preset annotation tool;
[0031] The converted recognition results are imported into the preset annotation tool, and the static map corresponding to each point cloud data subset is annotated using the preset annotation tool to obtain the annotation results corresponding to each point cloud data subset.
[0032] In a second aspect, the present application further provides a map element annotation device, the device comprising:
[0033] an acquisition module, configured to acquire video data and point cloud data collected by the vehicle in a target driving scene, and divide the video data and the point cloud data according to a preset acquisition time or a preset acquisition distance to obtain a plurality of video data subsets and a plurality of point cloud data subsets;
[0034] an identification module, configured to identify dynamic targets and static targets in each of the plurality of point cloud data subsets, remove point clouds corresponding to the dynamic targets, and perform coordinate transformation on point clouds corresponding to the static targets, to obtain a static map corresponding to each point cloud data subset;
[0035] a fusion and recognition module, configured to perform multimodal fusion of the static map and the video data to obtain a bird's-eye view corresponding to each point cloud data subset, and input the bird's-eye view into a pre-trained map element recognition model to perform map element recognition and obtain a recognition result;
[0036] The labeling module is used to label the static map corresponding to each point cloud data subset using the recognition result to obtain the labeling result corresponding to each point cloud data subset.
[0037] In a third aspect, the present application further provides an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;
[0038] Memory for storing computer programs;
[0039] The processor is configured to implement the map element labeling method described in any one of the first aspects when executing the program stored in the memory.
[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the map element annotation method described in any one of the first aspects.
[0041] The above-mentioned technical solution provided by the present application has the following advantages over the prior art: the method provided by the present application obtains video data and point cloud data collected by the vehicle in the target driving scene, and divides the video data and the point cloud data according to a preset collection time or a preset collection distance to obtain multiple video data subsets and multiple point cloud data subsets; identifies dynamic targets and static targets in each point cloud data subset of the multiple point cloud data subsets, removes the point clouds corresponding to the dynamic targets, and performs coordinate transformation on the point clouds corresponding to the static targets to obtain a static map corresponding to each point cloud data subset; performs multimodal fusion of the static map and the video data to obtain a bird's-eye view corresponding to each point cloud data subset, and inputs the bird's-eye view into a pre-trained map element recognition model to perform map element recognition to obtain a recognition result; uses the recognition result to annotate the static map corresponding to each point cloud data subset to obtain an annotation result corresponding to each point cloud data subset. Through this approach, a pre-trained map feature recognition model can be used to automatically identify and annotate map features in a static map, eliminating the need for manual annotation. This improves the efficiency and accuracy of map feature annotation. Furthermore, because the pre-trained map feature recognition model requires multimodal fusion of static map and video data before identifying map features, the accuracy and robustness of map feature recognition are improved, further enhancing the accuracy of map feature annotation. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0044] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.
[0045] Figure 1 A flowchart of a map element annotation method provided in an embodiment of the present application;
[0046] Figure 2A schematic diagram of a process for obtaining map element recognition results provided in an embodiment of the present application;
[0047] Figure 3 A schematic diagram of a process for constructing a static map provided in an embodiment of the present application;
[0048] Figure 4 A schematic diagram of a static map provided in an embodiment of the present application;
[0049] Figure 5 A schematic diagram of a process for marking map elements provided in an embodiment of the present application;
[0050] Figure 6 A schematic diagram of the structure of a map element annotation device provided in an embodiment of the present application;
[0051] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0052] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0053] The disclosure below provides many different embodiments or examples for implementing different configurations of the present invention. To simplify the disclosure of the present invention, the components and configurations of specific examples are described below. Of course, these are merely examples and are not intended to limit the present invention. In addition, the present invention may repeat reference numerals and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or configurations discussed.
[0054] In order to solve the problem that map elements are usually labeled manually in the prior art, which has low labeling efficiency and low accuracy, the present application provides a map element labeling method, device, electronic device and storage medium, which can improve the efficiency and accuracy of map element labeling.
[0055] See also Figure 1 , Figure 1 This is a flow chart of a map element annotation method provided in an embodiment of the present application. Figure 1 As shown, the map element annotation method may include the following steps:
[0056] Step 101: Obtain video data and point cloud data collected by a vehicle in a target driving scene, and divide the video data and point cloud data according to a preset collection time or a preset collection distance to obtain multiple video data subsets and multiple point cloud data subsets.
[0057] Specifically, the target driving scenario can be any driving scenario, including daytime, nighttime, rainy or sunny, on highways, and in urban areas. The video data and point cloud data are acquired by the vehicle's perception system. As an optional implementation, the video data and point cloud data can be collected using a ground truth system acquisition vehicle. The sensors required for this ground truth system acquisition vehicle may include lidar, millimeter-wave radar, cameras, and real-time kinematic (RTK) positioning technology.
[0058] The above-mentioned preset acquisition time and the above-mentioned preset acquisition distance can be set according to actual needs, and are not specifically limited in the embodiments of the present application. As an optional implementation, the above-mentioned preset acquisition time can be 30 seconds, and the above-mentioned preset acquisition distance can be 300 meters. When dividing the video data and point cloud data, the preset acquisition time can be used for division, that is, the video data and point cloud data are divided into a section every 30 seconds, forming multiple video data subsets and multiple point cloud data subsets with time as the dimension; or, the preset acquisition distance can be used for division, that is, the video data and point cloud data are divided into a section every 300 meters of driving distance, forming multiple video data subsets and multiple point cloud data subsets with driving distance as the dimension. Each video data subset includes multiple continuous video frames, and each point cloud data subset includes multiple continuous point cloud frames.
[0059] It should be noted that when segmenting video data and point cloud data, you can manually choose whether to segment by preset collection duration or preset collection distance based on the target driving scene. For example, in congested urban areas, where traffic jams are common or driving speeds are slow, you can segment by preset collection distance; while in less congested highways, you can segment by preset collection duration.
[0060] Step 102: Identify dynamic targets and static targets in each of the multiple point cloud data subsets, remove the point clouds corresponding to the dynamic targets, and perform coordinate transformation on the point clouds corresponding to the static targets to obtain a static map corresponding to each point cloud data subset.
[0061] Specifically, the dynamic targets mentioned above refer to objects that move over time, such as vehicles and pedestrians, and the static targets mentioned above refer to objects that do not move over time, such as buildings and ground elements.
[0062] In this step, the targets in each frame of the point cloud image in each point cloud data subset can be identified, and dynamic targets and static targets can be identified based on the position changes or shape changes of the targets in each frame of the point cloud image. Then, the point clouds corresponding to the dynamic targets in each frame of the point cloud image can be removed, and then the point clouds corresponding to the static targets in each frame of the point cloud image can be coordinate transformed and spliced to obtain the static maps corresponding to each point cloud data subset.
[0063] Traditional simultaneous localization and mapping (SLAM) mapping methods have the following disadvantages: ① High computational complexity: Traditional SLAM methods typically use complex mathematical models and algorithms to estimate the camera's motion trajectory and environmental map, which can lead to high computational complexity, especially when dealing with large-scale scenes. ② Requires a large amount of computing resources: Due to their high computational complexity, traditional SLAM methods typically require a large amount of computing resources, including processor performance and memory. ③ Sensitivity to initial estimates: Traditional SLAM methods are sensitive to the accuracy and stability of initial estimates. Incorrect initial estimates can lead to inaccurate or failed subsequent mapping. ④ High sensor accuracy requirements: Traditional SLAM methods have high sensor accuracy requirements, such as the camera's intrinsic parameters and the accuracy of the inertial measurement unit (IMU). This can increase hardware costs and dependence on the environment.
[0064] In this application, a static map can be obtained by combining three-dimensional (3D) target detection and coordinate transformation, achieving accurate three-dimensional environment modeling. This provides good mapping data for the subsequent effective identification of map elements, without the interference of dynamic noise, and can more efficiently and accurately identify map elements, especially ground elements such as lane lines, curbs, and zebra crossings.
[0065] Step 103: Perform multimodal fusion on the static map and video data to obtain a bird's-eye view corresponding to each point cloud data subset, and input the bird's-eye view into a pre-trained map feature recognition model to perform map feature recognition and obtain a recognition result.
[0066] In order to address the problem of unclear dense point clouds caused by LiDAR in situations such as special materials, damaged roads or water accumulation, as well as the instability of LiDAR data calibration, and to improve the recall rate of annotations, semantic maps and texture maps are introduced in the embodiments of this application. These image types can provide more information and help improve the recognition accuracy of map elements. It should be noted that the semantic map and texture map here are images and texture maps obtained based on video data, respectively.
[0067] Texture maps can be calibrated using the extrinsic parameters of lidar and visual cameras, enabling pixel-level fusion to reconstruct road textures. These images provide rich information and, in theory, can be annotated as long as there is scene data related to driving conditions. Semantic maps utilize the perception results obtained by fusion of lidar and cameras, and incorporating these perception results can address situations with weaker textures.
[0068] In this step, the static map and video data are multimodally fused to obtain a bird's-eye view (BEV) corresponding to each point cloud data subset. This BEV is then input into a pre-trained map feature recognition model to perform map feature recognition and obtain a recognition result. It should be noted that the pre-trained map feature recognition model can be any deep learning model or neural network model capable of map feature recognition, and this embodiment of the application does not impose any specific limitation.
[0069] Step 104: Use the recognition results to annotate the static map corresponding to each point cloud data subset to obtain the annotation results corresponding to each point cloud data subset.
[0070] After obtaining the recognition results of the pre-trained map feature recognition model, the recognition results can be directly used to annotate the static map corresponding to each point cloud data subset to obtain the annotation results corresponding to each point cloud data subset; or the recognition results can be first converted into a data structure suitable for a preset annotation tool, and then the converted recognition results can be annotated to obtain the annotation results corresponding to each point cloud data subset.
[0071] In this embodiment, a pre-trained map element recognition model can be used to automatically identify and annotate map elements in a static map, eliminating the need for manual annotation. This improves the efficiency and accuracy of map element annotation. Furthermore, because the pre-trained map element recognition model requires multimodal fusion of static map and video data before identifying map elements, the accuracy and robustness of map element recognition are improved, further enhancing the accuracy of map element annotation.
[0072] Furthermore, the map feature recognition model includes a feature extractor, a regressor, and a classifier;
[0073] In step 103, the bird's-eye view is input into a pre-trained map feature recognition model to perform map feature recognition, and a recognition result is obtained, including:
[0074] The bird's-eye view is input into the feature extractor for feature extraction to obtain the feature information corresponding to each map element in the target driving scene;
[0075] Input the feature information corresponding to each map element into the regressor for regression processing to obtain the geometric attributes corresponding to each map element, and input the feature information corresponding to each map element into the classifier for classification processing to obtain the type corresponding to each map element;
[0076] The geometric attributes corresponding to each map element and the type corresponding to each map element are determined as the recognition result.
[0077] In one embodiment, the structure of the map element recognition model is as follows: Figure 2 As shown, it can include a feature extractor, a regressor, and a classifier. The feature extractor is built based on the Transformer network, which effectively captures complex relationships between input data and extracts semantically informative feature representations. It should be noted that in a standard Transformer network, the self-attention mechanism requires calculating the attention weight of each map feature with respect to all map features in the sequence, resulting in time and space complexity proportional to the square of the sequence length. To reduce computational and memory requirements, this application proposes a decoupled self-attention mechanism that decomposes self-attention into two steps: the calculation of the query (Q), the key (K), and the weighted summation of the values (V). This decomposition allows the model to process the query and key independently, and then only perform a weighted summation on the values. In the hierarchical query mechanism, the model generates a set of queries (Q) at different levels, each focusing on information of a different scope or granularity. Each query level independently calculates the attention score with respect to all keys (K), but only sums the corresponding values (V). Assume there is a sequence X, which is divided into multiple queries Q and keys K, and corresponding values V, then it can be expressed as follows:
[0078] ;
[0079] ;
[0080] ;
[0081] in, 、 and is a learnable weight matrix.
[0082] The decoupled self-attention mechanism can be expressed as follows:
[0083] ;
[0084] in, is the dimension of the key K, ensuring that the scaling factor is proportional to the dimension. Represents the transposed matrix of key K.
[0085] Through this layered and decoupled approach, the map feature recognition model can process data more efficiently, because queries at each level only need to calculate attention scores with all keys, while the weighted sum of values can be performed independently. This approach reduces unnecessary calculations and memory usage, especially for long sequence data, and can significantly improve the efficiency of the model.
[0086] After extracting features using a feature extractor, regressors and classifiers can be used to identify and segment each map element. Regressors are used to infer the geometric properties of each map element, such as location and shape, while classifiers are used to determine the type of each map element, such as lane lines and curbs. Finally, the recognition results of each map element are presented in segments on the static map. This means that the static map is divided into different areas, and the identified map elements, such as lane lines, curbs, and zebra crossings, are marked in each area. This makes the static map clearer and easier to read, facilitating subsequent application and analysis.
[0087] Through the above method, the map element recognition model can be used to accurately identify each map element, which is convenient for subsequent map element labeling.
[0088] Furthermore, the above step of inputting the bird's-eye view into the feature extractor for feature extraction to obtain feature information corresponding to each map element includes:
[0089] Adjust the attention weight corresponding to each map element;
[0090] The bird's-eye view is input into the feature extractor, and the adjusted attention weights corresponding to each map element are used for feature extraction to obtain the feature information corresponding to each map element.
[0091] In one embodiment, an attention mechanism is introduced during the feature extraction process using a feature extractor to enhance the model's focus on specific map elements. Therefore, the attention weights corresponding to each map element can be adjusted according to project requirements. The bird's-eye view is then input into the feature extractor, and feature extraction is performed using the adjusted attention weights for each map element to obtain feature information corresponding to each map element. For example, by assigning different attention weights to features in different locations, the model can be more focused on ground element areas, thereby improving ground element recognition accuracy.
[0092] Furthermore, before step 103, inputting the bird's-eye view into a pre-trained map feature recognition model to perform map feature recognition and obtaining a recognition result, the method further includes:
[0093] Iteratively training a model to be trained using a first prediction branch and a second prediction branch, respectively, wherein the data input to the model to be trained by the first prediction branch is a bird's-eye view sample, and the data input to the model to be trained by the second prediction branch is a positive sample and a plurality of negative samples paired with the positive sample, and the model to be trained includes a feature extractor, a regressor, and a classifier;
[0094] The loss value of the model to be trained is calculated using a preset loss function, and when the loss value is lower than a preset threshold, the iterative training of the model to be trained is stopped to obtain a map feature recognition model, where the loss value is the sum of the loss value of the first prediction branch and the loss value of the second prediction branch.
[0095] In one embodiment, before using a map feature recognition model for map feature recognition, the model must first be trained. Specifically, the first prediction branch and the second prediction branch can each perform iterative training on the model to be trained. During the iterative training process, a loss value of the model to be trained is calculated using a preset loss function. When the loss value falls below a preset threshold, iterative training of the model to be trained is stopped, thereby obtaining the map feature recognition model. The preset threshold can be set based on actual conditions and is not specifically defined herein.
[0096] Because map feature recognition involves fewer samples of these features relative to the entire point cloud, there is an imbalance in the distribution of positive and negative samples, potentially leading to slow convergence when training the model. To address this issue, an additional one-to-many prediction branch (i.e., the second prediction branch) is introduced to increase the number of positive samples, helping the model learn faster and accelerating convergence. To address the sample imbalance, an auxiliary one-to-many prediction branch is designed. This branch pairs a single positive sample with multiple negative samples, thereby increasing the number of positive samples represented. In the model, in addition to the main branch (i.e., the first prediction branch) responsible for conventional classification or regression, the additional one-to-many prediction branch focuses on learning how to distinguish positive samples from a background composed of multiple negative samples. During training, the model's loss function includes not only the loss of the main branch but also the loss of the one-to-many prediction branch. This forces the model to optimize both components simultaneously during training.
[0097] As an optional implementation, the preset loss function can be expressed using the following formula:
[0098] ;
[0099] in, Represents the total loss value, represents the loss value of the first prediction branch, represents the loss value of the second prediction branch, Represents the weight parameter. Used to balance the contribution of the loss value of the first prediction branch and the loss value of the second prediction branch, the loss value of the first prediction branch Is the cross entropy loss, mean square error loss. The loss value of the second prediction branch It is a binary classification loss used to distinguish a single positive sample from multiple negative samples.
[0100] By optimizing the preset loss function, the model to be trained can simultaneously learn the characteristics of the main task and the auxiliary task, thereby improving learning efficiency and convergence speed.
[0101] Furthermore, the above step 103, performing multimodal fusion of the static map and the video data to obtain a bird's-eye view corresponding to each point cloud data subset, includes:
[0102] Generate dense point cloud data based on static maps, and generate texture maps based on video data;
[0103] Spatial alignment of dense point cloud data and texture maps;
[0104] The spatially aligned dense point cloud data and texture map are multimodally fused to obtain a bird's-eye view.
[0105] In one embodiment, see Figure 2 When performing multimodal fusion of static maps and video data, dense point cloud data can be generated based on the static map, and a texture map can be generated based on the video data. The dense point cloud data and the texture map are then spatially aligned, and the spatially aligned dense point cloud data and the texture map are multimodally fused to obtain a bird's-eye view.
[0106] It should be noted that in addition to static maps and video data, IMU data is also needed for attitude correction and motion compensation, so that the dense point cloud data and texture map can be spatially aligned to ensure the accuracy of the position and shape of map elements in the bird's-eye view, which is conducive to the fusion of the generated dense point cloud and texture map to form a bird's-eye view, which facilitates the subsequent use of the map element recognition model to identify map elements in the bird's-eye view.
[0107] Furthermore, the above step 102 of identifying dynamic objects and static objects in each of the plurality of point cloud data subsets includes:
[0108] Obtaining target information in each frame of point cloud images in a target point cloud data subset, wherein the target point cloud data subset is any point cloud data subset in a plurality of point cloud data subsets, and the target point cloud data subset includes a plurality of frame point cloud images;
[0109] Generate a target list corresponding to each frame of the point cloud image based on the target information, where the target list is used to record the target type, position, size and motion state contained in each frame of the point cloud image;
[0110] Based on the target list corresponding to each frame of the point cloud image, dynamic targets and static targets in the target point cloud data subset are identified.
[0111] In one embodiment, target information in each frame point cloud image in the target point cloud data subset can be obtained, and then a target list corresponding to each frame point cloud image is generated based on the target information. Then, based on the target list corresponding to each frame point cloud image, dynamic targets and static targets in the target point cloud data subset can be identified.
[0112] Specifically, if Figure 3 As shown in the figure, we can first detect the target in each frame point cloud image, and generate a target list for each frame point cloud image to record the category, position, size, motion state and other information of each target. Track dynamic targets in consecutive frame point cloud images to identify their motion patterns, and use tracking algorithms (such as Kalman filter or optical flow method, etc.) to associate dynamic targets detected across frames, obtain the corresponding points of dynamic targets in different frames, and prepare for the point cloud removal of dynamic targets. For the point cloud in each frame point cloud image, the target list is used to record the category, position, size and motion state of each target. and a dynamic target set Dt, perform the following operations:
[0113] ;
[0114] in, represents the static point cloud of the tth frame in the point cloud data subset, Represents the point cloud image of the tth frame in the point cloud data subset, represents a set of dynamic targets in a subset of point cloud data, Represents a dynamic target, Represents a set, A collection of point clouds representing all dynamic objects in a subset of point cloud data.
[0115] The IMU data is used to compensate for the dynamic motion of the ego vehicle to reduce motion distortion in the point cloud. The motion of the ego vehicle between consecutive frames is estimated using the IMU data, and the corresponding pose transformation matrix is calculated. It can be obtained by integrating the IMU data using the following formula:
[0116] ;
[0117] Among them, R represents the rotation matrix of the pose of the vehicle in each frame of the point cloud image, t represents the translation vector of the pose of the vehicle in each frame of the point cloud image, and Indicates the lower and upper limits of the time interval.
[0118] Taking into account the dynamic compensation, update the transformation of the vehicle pose to the first frame coordinate system. The point p in the point cloud can be transformed by the coordinate Transform to the new coordinate system:
[0119] ;
[0120] in, represents the pose transformation matrix, represents the static point cloud of the tth frame in the point cloud data subset, Represents the static point cloud of the tth frame after coordinate transformation in the point cloud data subset.
[0121] Finally, merge the point clouds of each frame in the point cloud data subset. This can be done by the following operations:
[0122] ;
[0123] in, Represents a static map corresponding to a subset of point cloud data, represents the static point cloud of the tth frame after coordinate transformation in the point cloud data subset, and N is the number of frames in the point cloud data subset.
[0124] The final static map is as follows Figure 4 As shown in Figure 2, map features such as lane lines, zebra crossings, etc. can be identified in dense point clouds (in Figure 4 The black part in the middle represents map elements such as lane lines and zebra crossings, and the gray part represents other elements such as buildings), providing a reliable data source for subsequent model detection and manual annotation.
[0125] Furthermore, in step 104, the static map corresponding to each point cloud data subset is annotated using the recognition result to obtain the annotation result corresponding to each point cloud data subset, including:
[0126] Convert the recognition results into the data structure required by the preset annotation tool;
[0127] The converted recognition results are imported into the preset annotation tool, and the preset annotation tool is used to annotate the static map corresponding to each point cloud data subset to obtain the annotation results corresponding to each point cloud data subset.
[0128] In one embodiment, when annotating the static map corresponding to each point cloud data subset, the recognition results can be converted into the data structure required by a preset annotation tool. The converted recognition results are then imported into the preset annotation tool, and the static map corresponding to each point cloud data subset is annotated using the preset annotation tool to obtain the annotation results corresponding to each point cloud data subset. Specifically, the recognition results can be aligned with the format required by the preset annotation tool, i.e., the recognition results are converted into a data structure suitable for the preset annotation tool to ensure correct import and processing. The aligned recognition results are then imported into the preset annotation tool for pre-annotation. The preset annotation tool is software specifically designed for processing point cloud data, with a convenient visual interface and annotation functionality. By importing the recognition results into the preset annotation tool, each map element can be easily viewed and edited. Finally, the annotation results are manually corrected and verified to ensure the accuracy and completeness of the labels. Manual corrections can include adding missing map elements, deleting incorrect labels, and adjusting position or shape. Through manual intervention, the quality and reliability of map element annotation can be further improved. The manually corrected annotation results are confirmed as the true values of the labels. These labels can be used for tasks such as training models and evaluating algorithm performance, and are the basis for map data processing and analysis.
[0129] The map element annotation method provided in the embodiment of the present application can identify map elements and annotate map elements by using video clips (divided by time or distance), SLAM mapping technology, and multimodal data (point cloud images, texture images, and pictures). By integrating information from different modalities, the accuracy and robustness of map element recognition are improved, providing strong support for subsequent map processing and applications. The specific process is as follows: Figure 5 Therefore, the map element annotation method provided by the embodiment of the present application has the following beneficial effects:
[0130] 1. Considering the temporal dimension: Traditional map feature annotation methods often only consider spatial information, ignoring the importance of the temporal dimension. However, for dynamic scenes or objects that change over time, such as vehicles and pedestrians, their positions and states may vary at different points in time. Therefore, introducing the temporal dimension can more accurately capture the movement and changes of these objects.
[0131] 2. Use historical data for prediction: By analyzing historical data, patterns or regularities can be discovered, allowing predictions about future data. In 4D pre-annotation, existing time series data can be used for predictions. The predictions are then used as the initial values for annotation, followed by manual correction and verification. This significantly reduces the workload of manual annotation and improves both efficiency and accuracy.
[0132] 3. Pre-labeling future data: In some scenarios, data for a period of time in the future can be labeled in advance. For example, in traffic scenarios, historical data and traffic rules can be used to predict the location and behavior of vehicles and pedestrians in the next few seconds or minutes. Labeling can then be performed based on these predictions, thus obtaining labels for future data in advance and saving the time and cost of subsequent labeling.
[0133] 4. Enhanced Data Utilization Value: 4D pre-labeling not only saves labeling time and costs but also enhances data utilization. Labeled temporal information provides additional information for subsequent data analysis and algorithm development, such as behavioral analysis and trajectory prediction. Labeled time series data can also be used for model training, improving performance in dynamic scenarios.
[0134] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a map element annotation device provided in an embodiment of the present application. Figure 6 As shown, the map element annotation device 600 includes:
[0135] An acquisition module 601 is configured to acquire video data and point cloud data collected by a vehicle in a target driving scene, and to divide the video data and point cloud data according to a preset acquisition time or a preset acquisition distance to obtain multiple video data subsets and multiple point cloud data subsets;
[0136] an identification module 602 for identifying dynamic targets and static targets in each of the plurality of point cloud data subsets, removing point clouds corresponding to dynamic targets, and performing coordinate transformation on point clouds corresponding to static targets to obtain a static map corresponding to each point cloud data subset;
[0137] The fusion and recognition module 603 is used to perform multimodal fusion of the static map and video data to obtain a bird's-eye view corresponding to each point cloud data subset, and input the bird's-eye view into a pre-trained map feature recognition model to perform map feature recognition and obtain a recognition result;
[0138] The labeling module 604 is used to label the static map corresponding to each point cloud data subset using the recognition result to obtain the labeling result corresponding to each point cloud data subset.
[0139] Furthermore, the map feature recognition model includes a feature extractor, a regressor, and a classifier;
[0140] The fusion and recognition module 603 includes:
[0141] The extraction submodule is used to input the bird's-eye view into the feature extractor for feature extraction, and obtain the feature information corresponding to each map element in the target driving scene;
[0142] The processing submodule is used to input the feature information corresponding to each map element into the regressor for regression processing to obtain the geometric attributes corresponding to each map element, and input the feature information corresponding to each map element into the classifier for classification processing to obtain the type corresponding to each map element;
[0143] The determination submodule is used to determine the geometric attributes corresponding to each map element and the type corresponding to each map element as the recognition result.
[0144] Furthermore, the extraction submodule includes:
[0145] An adjustment unit, used to adjust the attention weight corresponding to each map element;
[0146] The extraction unit is used to input the bird's-eye view into the feature extractor, and use the adjusted attention weights corresponding to each map element to perform feature extraction to obtain feature information corresponding to each map element.
[0147] Furthermore, the map element annotation device 600 further includes:
[0148] a training module, configured to iteratively train a model to be trained using a first prediction branch and a second prediction branch, respectively, wherein the data input to the model to be trained by the first prediction branch is a bird's-eye view sample, and the data input to the model to be trained by the second prediction branch is a positive sample and a plurality of negative samples paired with the positive sample, and the model to be trained includes a feature extractor, a regressor, and a classifier;
[0149] The calculation module is used to calculate the loss value of the model to be trained using a preset loss function, and when the loss value is lower than a preset threshold, stop iterative training of the model to be trained to obtain a map feature recognition model, wherein the loss value is the sum of the loss value of the first prediction branch and the loss value of the second prediction branch.
[0150] Furthermore, the fusion and recognition module 603 further includes:
[0151] The first generation submodule is used to generate dense point cloud data based on the static map and generate a texture map based on the video data;
[0152] Alignment submodule, used to spatially align dense point cloud data and texture maps;
[0153] The fusion submodule is used to perform multimodal fusion of the spatially aligned dense point cloud data and texture map to obtain a bird's-eye view.
[0154] Furthermore, the identification module 602 includes:
[0155] an acquisition submodule, configured to acquire target information in each frame of a point cloud image in a target point cloud data subset, wherein the target point cloud data subset is any point cloud data subset in a plurality of point cloud data subsets, and the target point cloud data subset includes a plurality of frame point cloud images;
[0156] The second generation submodule is used to generate a target list corresponding to each frame of the point cloud image according to the target information, wherein the target list is used to record the target type, position, size and motion state contained in each frame of the point cloud image;
[0157] The recognition submodule is used to identify dynamic targets and static targets in the target point cloud data subset based on the target list corresponding to each frame of the point cloud image.
[0158] Furthermore, the marking module 604 includes:
[0159] The conversion submodule is used to convert the recognition results into the data structure required by the preset annotation tool;
[0160] The annotation submodule is used to import the converted recognition results into the preset annotation tool, and use the preset annotation tool to annotate the static map corresponding to each point cloud data subset to obtain the annotation results corresponding to each point cloud data subset.
[0161] It should be noted that the map element annotation device 600 can implement the map element annotation method provided by any of the aforementioned method embodiments and can achieve the same technical effects, which will not be described in detail here.
[0162] like Figure 7 As shown, the embodiment of the present application further provides an electronic device, including a processor 711, a communication interface 712, a memory 713 and a communication bus 714, wherein the processor 711, the communication interface 712, and the memory 713 communicate with each other through the communication bus 714.
[0163] Memory 713, for storing computer programs;
[0164] In one embodiment of the present application, the processor 711 is configured to implement the map element annotation method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 713 .
[0165] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the map element annotation method provided by any one of the aforementioned method embodiments is implemented.
[0166] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.
[0167] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a general hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.
[0168] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "comprise", "include", "contain" and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the specific order described or illustrated, unless the order of execution is clearly indicated. It should also be understood that additional or alternative steps may be used.
[0169] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is intended to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A map element annotation method, characterized in that: The method comprises: Acquire video data and point cloud data collected by the vehicle in a target driving scene, and divide the video data and the point cloud data according to a preset collection time or a preset collection distance to obtain multiple video data subsets and multiple point cloud data subsets; Identify dynamic targets and static targets in each of the multiple point cloud data subsets, remove the point cloud corresponding to the dynamic target, and perform coordinate transformation on the point cloud corresponding to the static target to obtain a static map corresponding to each point cloud data subset; Performing multimodal fusion on the static map and the video data to obtain a bird's-eye view corresponding to each point cloud data subset, and inputting the bird's-eye view into a pre-trained map element recognition model to perform map element recognition to obtain a recognition result; Using the recognition results, annotating the static map corresponding to each point cloud data subset to obtain the annotation results corresponding to each point cloud data subset; The step of performing multimodal fusion of the static map and the video data to obtain a bird's-eye view corresponding to each point cloud data subset includes: generating dense point cloud data according to the static map, and generating a texture map according to the video data; spatially aligning the dense point cloud data and the texture map; The spatially aligned dense point cloud data and the texture map are multimodally fused to obtain the bird's-eye view.
2. The map element annotation method according to claim 1, characterized in that: The map element recognition model includes a feature extractor, a regressor and a classifier; The step of inputting the bird's-eye view into a pre-trained map element recognition model to perform map element recognition and obtain a recognition result includes: Inputting the bird's-eye view into the feature extractor for feature extraction to obtain feature information corresponding to each map element in the target driving scene; Inputting the feature information corresponding to each map element into the regressor for regression processing to obtain the geometric attributes corresponding to each map element, and inputting the feature information corresponding to each map element into the classifier for classification processing to obtain the type corresponding to each map element; The geometric attributes corresponding to each map element and the type corresponding to each map element are determined as the recognition result.
3. The map element annotation method according to claim 2, characterized in that: The step of inputting the bird's-eye view into the feature extractor for feature extraction to obtain feature information corresponding to each map element includes: Adjust the attention weights corresponding to each map element; The bird's-eye view is input into the feature extractor, and feature extraction is performed using the adjusted attention weights corresponding to each map element to obtain feature information corresponding to each map element.
4. The map element annotation method according to any one of claims 1 to 3, characterized in that: Before inputting the bird's-eye view into a pre-trained map element recognition model to perform map element recognition and obtain a recognition result, the method further includes: Iteratively training the model to be trained is performed using a first prediction branch and a second prediction branch, respectively, wherein the data input to the model to be trained by the first prediction branch is a bird's-eye view sample, the data input to the model to be trained by the second prediction branch is a positive sample and a plurality of negative samples paired with the positive sample, and the model to be trained includes a feature extractor, a regressor, and a classifier; The loss value of the model to be trained is calculated using a preset loss function, and when the loss value is lower than a preset threshold, the iterative training of the model to be trained is stopped to obtain the map element recognition model, wherein the loss value is the sum of the loss value of the first prediction branch and the loss value of the second prediction branch.
5. The map element annotation method according to any one of claims 1 to 3, characterized in that: The identifying of dynamic targets and static targets in each of the plurality of point cloud data subsets comprises: Acquire target information in each frame of point cloud images in a target point cloud data subset, wherein the target point cloud data subset is any point cloud data subset in the multiple point cloud data subsets, and the target point cloud data subset includes multiple frames of point cloud images; Generate a target list corresponding to each frame of the point cloud image according to the target information, wherein the target list is used to record the target type, position, size and motion state contained in each frame of the point cloud image; Based on the target list corresponding to each frame of the point cloud image, dynamic targets and static targets in the target point cloud data subset are identified.
6. The map element annotation method according to any one of claims 1 to 3, characterized in that: The step of labeling the static map corresponding to each point cloud data subset using the recognition result to obtain the labeling result corresponding to each point cloud data subset includes: Converting the recognition result into a data structure required by a preset annotation tool; The converted recognition results are imported into the preset annotation tool, and the static map corresponding to each point cloud data subset is annotated using the preset annotation tool to obtain the annotation results corresponding to each point cloud data subset.
7. A map element annotation device, characterized in that: The device comprises: An acquisition module, used to acquire video data and point cloud data collected by the vehicle in a target driving scene, and divide the video data and the point cloud data according to a preset acquisition time or a preset acquisition distance to obtain multiple video data subsets and multiple point cloud data subsets; an identification module, used to identify dynamic targets and static targets in each of the plurality of point cloud data subsets, remove the point cloud corresponding to the dynamic target, and perform coordinate transformation on the point cloud corresponding to the static target, so as to obtain a static map corresponding to each point cloud data subset; A fusion and recognition module, used for performing multimodal fusion of the static map and the video data to obtain a bird's-eye view corresponding to each point cloud data subset, and inputting the bird's-eye view into a pre-trained map element recognition model to perform map element recognition to obtain a recognition result; A labeling module, used to label the static map corresponding to each point cloud data subset using the recognition result, to obtain the labeling result corresponding to each point cloud data subset; Wherein, the fusion and recognition module includes: A first generating submodule, used to generate dense point cloud data according to the static map, and to generate a texture map according to the video data; An alignment submodule, used for spatially aligning the dense point cloud data and the texture map; The fusion submodule is used to perform multimodal fusion on the spatially aligned dense point cloud data and the texture map to obtain the bird's-eye view.
8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the map element labeling method described in any one of claims 1 to 6 when executing the program stored in the memory.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the map element labeling method according to any one of claims 1 to 6 is implemented.
