Labeling method and device, equipment and storage medium

Abstract

The invention provides an annotation method and device, equipment and a storage medium. In one example, the method comprises the steps of obtaining N groups of images collected by a target camera and N frames of laser point clouds collected by a laser radar; feature fusion is carried out on the N groups of images and the N frames of laser point clouds, and speed prediction is carried out on laser points in the N frames of laser point clouds according to fused features; according to the speed of each laser point in the N frames of laser point clouds and the acquisition time of each frame of laser point clouds, each laser point in the N frames of laser point clouds is mapped to a frame of laser point clouds at a specified moment; marking the laser point cloud at the specified moment to obtain a marking result of the laser point cloud at the specified moment; and according to a mapping relationship between each laser point in the N frames of laser point clouds and each laser point in the laser point clouds at the specified moment, mapping a labeling result of the laser point clouds at the specified moment back to the N frames of laser point clouds. The method can improve the labeling efficiency.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a marking method, device, equipment and storage medium. Background Art

[0002] In intelligent driving perception tasks, object detection is very important. BEV (Birds Eye View) object detection is a more mainstream object detection solution in intelligent driving perception tasks.

[0003] In the BEV target detection solution, a single or multiple cameras (or a multi-dimensional solution integrated with millimeter-wave radar or lidar) are used to map features to the BEV space and obtain 3D detection targets.

[0004] However, the BEV target detection scheme is highly dependent on data annotation. How to efficiently implement data annotation has become a hot research direction in recent years. Summary of the invention

[0005] In view of this, the present application provides a marking method, apparatus, device and storage medium.

[0006] Specifically, the present application is implemented through the following technical solutions: According to a first aspect of an embodiment of the present application, a marking method is provided, including: Obtain N groups of images captured by the target camera and N frames of laser point clouds captured by the laser radar; wherein the target camera includes K vehicle-mounted cameras, the total field of view of the K vehicle-mounted cameras covers the four sides of the vehicle body, the scanning range of the laser radar covers the four sides of the vehicle body, and the at least two vehicle-mounted cameras synchronously perform image acquisition; N≥2, K≥2, a group of images includes K frames of images acquired synchronously, the N groups of images correspond one-to-one to the N frames of laser point clouds, and the corresponding images are consistent with the acquisition time of the laser point clouds; Performing feature fusion on the N groups of images and the N frames of laser point clouds, and predicting the speed of the laser points in the N frames of laser point clouds according to the fusion features to obtain the speed of each laser point in the N frames of laser point clouds; According to the speed of each laser point in the N frames of laser point cloud and the acquisition time of each frame of laser point cloud, each laser point in the N frames of laser point cloud is mapped to a frame of laser point cloud at a specified time to obtain the laser point cloud at the specified time; Annotating the laser point cloud at the specified time to obtain an annotation result of the laser point cloud at the specified time; According to the mapping relationship between each laser point in the N frames of laser point cloud and each laser point in the laser point cloud at the specified time, the annotation result of the laser point cloud at the specified time is mapped back to the N frames of laser point cloud to obtain the annotation result of the N frames of laser point cloud.

[0007] According to a second aspect of an embodiment of the present application, there is provided a marking device, including: An acquisition unit is used to acquire N groups of images acquired by a target camera and N frames of laser point clouds acquired by a laser radar; wherein the target camera includes K vehicle-mounted cameras, the total field of view of the K vehicle-mounted cameras covers the four sides of the vehicle body, the scanning range of the laser radar covers the four sides of the vehicle body, and the at least two vehicle-mounted cameras synchronously acquire images; N≥2, K≥2, a group of images includes K frames of images acquired synchronously, the N groups of images correspond one-to-one to the N frames of laser point clouds, and the corresponding images are consistent with the acquisition time of the laser point clouds; A feature fusion unit, used for performing feature fusion on the N groups of images and the N frames of laser point clouds; A speed prediction unit, used for predicting the speed of the laser points in the N frames of laser point cloud according to the fusion features, to obtain the speed of each laser point in the N frames of laser point cloud; A mapping unit, configured to map each laser point in the N frames of laser point cloud to a frame of laser point cloud at a specified time according to the speed of each laser point in the N frames of laser point cloud and the acquisition time of each frame of laser point cloud, so as to obtain the laser point cloud at the specified time; A labeling unit, used to label the laser point cloud at the specified time to obtain a labeling result of the laser point cloud at the specified time; The mapping unit is further used to map the annotation result of the laser point cloud at the specified moment back to the N frames of laser point cloud according to the mapping relationship between each laser point in the N frames of laser point cloud and each laser point in the laser point cloud at the specified moment, so as to obtain the annotation result of the N frames of laser point cloud.

[0008] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method provided in the first aspect.

[0009] According to a fourth aspect of an embodiment of the present application, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0010] The technical solution provided by this application can at least bring the following beneficial effects: By acquiring N groups of images acquired by the target camera and N frames of laser point clouds acquired by the laser radar, feature fusion is performed on the acquired N groups of images and N frames of laser point clouds, and the speed of the laser points in the N frames of laser point clouds is predicted based on the fusion features to obtain the speed of each laser point in the N frames of laser point clouds. Based on the speed of each laser point in the N frames of laser point clouds and the acquisition time of each frame of laser point clouds, each laser point in the N frames of laser point clouds is mapped to a frame of laser point clouds at a specified time to obtain the laser point cloud at the specified time. Then, the laser point cloud at the specified time is annotated to obtain the annotation result of the laser point cloud at the specified time. The mapping relationship between each laser point in the frame laser point cloud and each laser point in the laser point cloud at the specified time, the annotation result of the laser point cloud at the specified time is mapped back to the N-frame laser point cloud, and the annotation result of the N-frame laser point cloud is obtained. By adopting the speed prediction technology of laser and image fusion, the speed of all laser points is predicted, so that all laser points are accumulated to the same frame laser point cloud at the specified time, and the frame laser point cloud is annotated. Then, according to the laser point mapping relationship, the annotation result is mapped back to the obtained N-frame laser point cloud, which improves the annotation efficiency. It is no longer limited to annotating static targets, but can realize the annotation of dynamic and static targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a flowchart of a marking method shown in an exemplary embodiment of the present application; Figure 2 is a schematic diagram of a sensor arrangement shown in an exemplary embodiment of the present application; Figure 3 This is a schematic diagram of a 4D annotation implementation process provided by an embodiment of the present application; Figure 4 This is a schematic diagram of a timing compensation implementation process provided by an embodiment of the present application; Figure 5 This is a schematic diagram of a speed prediction implementation process provided by an embodiment of the present application; Figure 6 This is a schematic diagram of a first-level fusion implementation process provided by an embodiment of the present application; Figure 7 This is a schematic diagram of a two-level fusion implementation process provided by an embodiment of the present application; Figure 8 This is a schematic diagram of a manual labeling implementation process provided by an embodiment of the present application; Fig. 9 This is a schematic diagram of a post-processing implementation flow of annotation provided in an embodiment of the present application; Fig.10 is a structural schematic diagram of a marking device provided in an embodiment of the present application; Fig.11It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0012] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0013] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0014] See also Figure 1 , is a flow chart of a marking method provided in an embodiment of the present application, such as Figure 1 As shown, the marking method may include the following steps: Step S100, obtaining N groups of images captured by the target camera and N frames of laser point cloud captured by the laser radar; wherein the target camera includes K vehicle-mounted cameras, the total field of view of the K vehicle-mounted cameras covers the surroundings of the vehicle body, and the scanning range of the laser radar covers the surroundings of the vehicle body; N≥2, K≥1, a group of images includes K frames of images captured synchronously, the N groups of images correspond one-to-one to the N frames of laser point cloud, and the corresponding images are consistent with the acquisition time of the laser point cloud.

[0015] In the embodiment of the present application, the above-mentioned target camera and laser radar refer to the target camera and laser radar installed on the vehicle in the intelligent driving scenario for assisting intelligent driving perception.

[0016] Exemplarily, the target camera may include K vehicle-mounted cameras, whose total field of view covers all around the vehicle body, and the scanning range of the lidar covers all around the vehicle body, so as to achieve comprehensive perception during the autonomous driving process.

[0017] Exemplarily, K≥1. In the case of K>1, the K vehicle-mounted cameras synchronously perform image acquisition.

[0018] For example, the target camera may include a total of four vehicle-mounted cameras installed at four positions on the front, back, left and right sides of the vehicle, and the lidar may be installed on the roof.

[0019] Exemplarily, the target camera and laser radar can simultaneously collect images and laser point clouds.

[0020] Among them, each camera among the multiple vehicle-mounted cameras included in the target camera also synchronously performs image acquisition.

[0021] In an embodiment of the present application, N groups of images captured by the target camera and N frames of laser point clouds captured by the lidar can be obtained respectively to implement subsequent annotation processing.

[0022] The N groups of images correspond one-to-one to the N frames of laser point clouds, and the corresponding images are collected at the same time as the laser point clouds.

[0023] It should be noted that, in the embodiment of the present application, the above-mentioned consistent acquisition time includes that the acquisition time is the same, or the difference between the acquisition times is within a tolerable error range.

[0024] Step S110, feature fusion is performed on N groups of images and N frames of laser point clouds, and the speed of the laser points in the N frames of laser point clouds is predicted based on the fusion features to obtain the speed of each laser point in the N frames of laser point clouds.

[0025] Step S120, mapping each laser point in N frames of laser point cloud to a frame of laser point cloud at a specified time according to the speed of each laser point in N frames of laser point cloud and the acquisition time of each frame of laser point cloud, to obtain the laser point cloud at the specified time.

[0026] In an embodiment of the present application, in order to improve the labeling efficiency, the speed of the laser points in the acquired laser point cloud can be predicted, and based on the speed of the laser points in the laser point cloud, the laser points in multiple frames of laser point clouds can be mapped to the same frame of laser point cloud. Thus, multi-frame laser point cloud labeling can be achieved by labeling the frame of laser point cloud.

[0027] Exemplarily, in order to realize the speed prediction of the laser points in the laser point cloud, the features of the N groups of images and the N frames of laser point clouds can be fused to obtain fused features, and then the speed of the laser points in the N frames of laser point clouds can be predicted based on the fused features.

[0028] When the speed of each laser point in N frames of laser point cloud is determined, each laser point in the N frames of laser point cloud can be mapped to a frame of laser point cloud at a specified time according to the speed of each laser point in the N frames of laser point cloud and the acquisition time of each frame of laser point cloud, so as to obtain the laser point cloud at the specified time.

[0029] Exemplarily, the designated time may be inconsistent with the acquisition time of each frame of the N frames of laser point cloud.

[0030] Step S130: annotate the laser point cloud at the specified time to obtain an annotation result of the laser point cloud at the specified time.

[0031] Step S140, according to the mapping relationship between each laser point in the N-frame laser point cloud and each laser point in the laser point cloud at the specified time, the annotation result of the laser point cloud at the specified time is mapped back to the N-frame laser point cloud to obtain the annotation result of the N-frame laser point cloud.

[0032] In the embodiment of the present application, when the laser point cloud at the specified time is obtained in the above manner, the laser point cloud at the specified time can be annotated to obtain the annotation result of the laser point cloud at the specified time.

[0033] For example, the laser point cloud at a specified moment may be labeled by manual labeling, or the laser point cloud at a specified moment may be labeled by model pre-labeling combined with manual labeling.

[0034] In an embodiment of the present application, the annotation result of the laser point cloud at the specified moment can be mapped back to the N frames of laser point cloud based on the annotation result of the laser point cloud at the specified moment, and the mapping relationship between each laser point in the N frames of laser point cloud and each laser point in the laser point cloud at the specified moment, so as to obtain the annotation result of the N frames of laser point cloud.

[0035] Exemplarily, when the annotation results of N frames of laser point clouds are obtained in the above manner, the obtained annotation results can also be fine-tuned, for example, by using a network form, such as pointnet (a deep learning architecture for processing point cloud data), or by using a RANSAC (Random Sample Consensus) algorithm, etc., to fine-tune the annotation results to achieve accurate annotation.

[0036] It can be seen that in Figure 1In the method flow shown, by acquiring N groups of images acquired by the target camera and N frames of laser point clouds acquired by the laser radar, feature fusion is performed on the acquired N groups of images and N frames of laser point clouds, and the speed of the laser points in the N frames of laser point clouds is predicted based on the fusion features to obtain the speed of each laser point in the N frames of laser point clouds, and based on the speed of each laser point in the N frames of laser point clouds and the acquisition time of each frame of laser point clouds, each laser point in the N frames of laser point clouds is mapped to a frame of laser point clouds at a specified time to obtain the laser point cloud at the specified time, and then the laser point cloud at the specified time is annotated to obtain the annotation result of the laser point cloud at the specified time. According to the mapping relationship between each laser point in the N-frame laser point cloud and each laser point in the laser point cloud at the specified time, the annotation result of the laser point cloud at the specified time is mapped back to the N-frame laser point cloud, and the annotation result of the N-frame laser point cloud is obtained. By adopting the speed prediction technology of laser and image fusion, the speed of all laser points is predicted, so that all laser points are accumulated to the same frame of laser point cloud at the specified time, and the frame of laser point cloud is annotated. Then, according to the laser point mapping relationship, the annotation result is mapped back to the obtained N-frame laser point cloud, which improves the annotation efficiency, and is no longer limited to the annotation of static targets, but can realize the annotation of dynamic and static targets.

[0037] In some embodiments, the feature fusion of N groups of images and N frames of laser point clouds may include: Perform 2D feature extraction on N groups of images, perform perspective conversion on the extracted image features to obtain the first BEV feature; perform 3D feature extraction on N frames of laser point cloud to obtain laser features; And, based on the conversion relationship between the image pixel coordinates and the laser radar coordinates, the N groups of images and the N frames of laser point clouds are first-level fused to obtain first-level fusion features, 2D features are extracted from the first-level fusion features, and the extracted fusion image features are subjected to perspective conversion to obtain second BEV features; The first BEV feature, the laser feature, and the second BEV feature are subjected to secondary feature fusion to obtain a final fused feature.

[0038] Exemplarily, in the process of feature fusion of the above-mentioned N groups of images and N frames of laser point clouds, on the one hand, 2D feature extraction can be performed on the N groups of images respectively, and the extracted image features can be visually converted to obtain BEV features (which can be called first BEV features); and 3D feature extraction can be performed on the N frames of laser point clouds to obtain laser features.

[0039] For example, 2D convolution / 2D sparse convolution may be used to extract features from the N groups of images, and then the features are converted into BEV features (i.e., first BEV features) through a perspective conversion network.

[0040] Exemplarily, the perspective conversion network can be in the form of LSS (Lift, Splat, Shoot, a technique for converting from camera perspective to BEV perspective) or transformer.

[0041] Exemplarily, the perspective conversion network can achieve visual conversion by performing a dot product operation on image features and depth distribution.

[0042] That is, for the image features obtained by performing 2D feature extraction on the above N groups of images, the image features can be dot-multiplied with the depth distribution, and a maximum pooling operation can be performed to achieve visual conversion and obtain the corresponding BEV features (i.e., the first BEV features mentioned above).

[0043] Exemplarily, for each pixel feature, the depth distribution is to predict its J (J ≥ 2) distance intervals and the probability of falling in each distance interval.

[0044] For laser point clouds, 3D convolution / 3D sparse convolution can be used to extract laser features.

[0045] It should be noted that for laser point clouds, before feature extraction, pillar (a point cloud encoding method) and other forms of preprocessing can be used, and 3D convolution / 3D sparse convolution can be used to extract features from the preprocessed laser point clouds.

[0046] On the other hand, based on the conversion relationship between the image pixel coordinates and the laser radar coordinates, the above N groups of images and N groups of laser point clouds can be fused at the first level to obtain the first level fusion features.

[0047] Exemplarily, the conversion relationship between image pixel coordinates and lidar coordinates can be calculated based on camera intrinsic parameters and external parameters of joint calibration of the camera and lidar.

[0048] In an example, based on the conversion relationship between the image pixel coordinates and the laser radar coordinates, the N groups of images and the N frames of laser point clouds are first-level fused to obtain the first-level fusion features, which may include: For any pixel point in any image in the N groups of images, based on the conversion relationship between the image pixel coordinates and the laser radar coordinates of the image, determine the laser point corresponding to the pixel point in the laser point cloud corresponding to the image; Performing 2D feature extraction on the pixel point to obtain a feature corresponding to the pixel point; and performing feature extraction on the laser point to obtain a feature corresponding to the laser point; The pixel point, the laser point, the feature corresponding to the pixel point, and the feature corresponding to the laser point are spliced ​​to obtain the corresponding first-level fusion feature.

[0049] Exemplarily, for any pixel point in any image in the above-mentioned N groups of images, the laser point corresponding to the pixel point in the laser point cloud corresponding to the image can be determined based on the conversion relationship between the image pixel coordinates and the laser radar coordinates of the image.

[0050] When the laser point corresponding to the pixel point is determined, on the one hand, 2D feature extraction can be performed on the pixel point to obtain the feature corresponding to the pixel point; on the other hand, feature extraction can be performed on the laser point to obtain the feature corresponding to the laser point.

[0051] For example, for a pixel point, it can be processed through 2D convolution and maxpooling network to obtain the features corresponding to the pixel point.

[0052] For the laser point, the features corresponding to the laser point can be obtained through MLP (Multi-Layer Perceptron) and maxpooling network processing.

[0053] It should be noted that, for the laser points in the laser point cloud, in the process of feature extraction, the laser points within a specified neighborhood range (such as a range with a radius of d) of the laser point can be determined, and the MLP and maxpooling network processing can be used to obtain the features corresponding to the laser points.

[0054] When the features corresponding to the pixel point and the features corresponding to the laser point are obtained in the above manner, the pixel point, the laser point, the features corresponding to the pixel point, and the features corresponding to the laser point can be spliced ​​to obtain the first-level fusion features corresponding to the pixel point.

[0055] Exemplarily, when a first-level fusion feature is obtained, 2D feature extraction can be performed on the first-level fusion feature, and visual transformation can be performed on the extracted fusion image feature to obtain a corresponding BEV feature (which can be called a second BEV feature).

[0056] Exemplarily, the visual transformation of the fused image features can be achieved by performing a dot product operation on the fused image features and the depth distribution, and performing a maximum pooling operation to obtain the second BEV feature.

[0057] Furthermore, the first BEV feature, the laser feature, and the second BEV feature may be subjected to secondary feature fusion to obtain a final fused feature.

[0058] In one example, performing secondary feature fusion on the first BEV feature, the laser feature, and the second BEV feature to obtain a final fused feature may include: Perform feature splicing on the image features and the fused image features, and perform a maximum pooling operation to obtain a spliced ​​fused feature, and perform a dot multiplication operation on the spliced ​​fused feature and the laser feature to obtain a laser fused feature; The laser fusion feature, the first BEV feature, and the second BEV feature are fused at the second level using the cross attention mechanism to obtain the final fused feature.

[0059] Exemplarily, in the secondary feature fusion process, on the one hand, the first BEV feature, the second BEV feature, and the laser feature can be obtained in the above manner.

[0060] On the other hand, the image features (the image features obtained by performing 2D feature extraction on the above-mentioned N groups of images) and the fused image features (the fused image features obtained by performing 2D feature extraction on the above-mentioned first-level fused features) can be fused to obtain corresponding fused features (which can be called spliced ​​fused features), and the spliced ​​fused features can be fused with the laser features to obtain laser fused features.

[0061] As an example, the above-mentioned image features and fused image features may be concatenated (concat), and a maxpooling operation may be performed to obtain concatenated fused features.

[0062] As an example, a dot product operation may be performed on the spliced ​​fusion feature and the laser feature to obtain the laser fusion feature.

[0063] It should be recognized that in the embodiments of the present application, the specific processing strategies adopted in the above-mentioned feature fusion are merely specific examples and do not constitute a limitation on the scope of protection of the present application. For example, feature fusion of image features and fused image features may also be performed by weighted summation or feature mapping, and its specific implementation will not be described in detail here.

[0064] Furthermore, the laser fusion feature, the first BEV feature, and the second BEV feature may be subjected to secondary fusion to obtain a final fusion feature.

[0065] As an example, a cross attention mechanism may be used to perform secondary fusion on the laser fusion feature, the first BEV feature, and the second BEV feature to obtain a final fusion feature.

[0066] In some embodiments, the solution provided by the embodiments of the present application may also include: Pillar processing is performed on N frames of laser point cloud to obtain processed N frames of laser point cloud.

[0067] The above-mentioned prediction of the speed of the laser points in the N-frame laser point cloud based on the fusion features to obtain the speed of each laser point in the N-frame laser point cloud may include: The speed of each pillar in N frames of laser point cloud is predicted based on the fusion features to obtain the speed of each pillar in N frames of laser point cloud; For any pillar, the velocity of the pillar is assigned to each laser point in the pillar.

[0068] Exemplarily, when pillar processing is performed on a laser point cloud, a pillar includes a laser point in a three-dimensional space of a preset size.

[0069] Exemplarily, when the fusion feature is determined in the above manner, in the process of predicting the speed of the laser point based on the fusion feature, the speed prediction can be performed in units of pillar.

[0070] That is, the speed of each pillar in N frames of laser point cloud can be predicted based on the fusion features to obtain the speed of each pillar in N frames of laser point cloud.

[0071] Exemplarily, for any pillar, the speed of the pillar can be assigned to each laser point in the pillar, thereby improving the efficiency of laser point speed prediction.

[0072] In some embodiments, mapping each laser point in N frames of laser point cloud to a frame of laser point cloud at a specified time according to the speed of each laser point in N frames of laser point cloud and the acquisition time of each frame of laser point cloud to obtain the laser point cloud at the specified time may include: For any laser point in any frame of laser point cloud, based on the speed of the laser point and the time difference between the acquisition time of the frame of laser point cloud and the specified time, determine the compensation amount corresponding to the laser point; According to the compensation amount corresponding to the laser point, the laser point is mapped to the laser point cloud at a specified time.

[0073] Exemplarily, when the speed of each laser point is determined, the compensation amount corresponding to the laser point can be determined based on the speed of the laser point and the time difference between the acquisition time of the laser point cloud and the specified time.

[0074] Exemplarily, for any laser point, the speed of the laser point may include speeds in three directions: X, Y, and Z, and the compensation amount corresponding to the laser point may also include compensation amounts in three directions: X, Y, and Z.

[0075] For example, assume that the i-th laser point of the laser point cloud collected at time T is ci_T, and its velocities are ci_T_vx, ci_T_vy, and ci_T_vz. The specified time is time T-1, and the time difference between time T and time T-1 is t. Then the compensation amount corresponding to the laser point is: X = ci_T_vx * t Y = ci_T_vy * t Z = ci_T_vz * t Exemplarily, for any laser point, when the compensation amount corresponding to the laser point is determined, the laser point can be mapped to the laser point cloud at a specified time according to the compensation amount corresponding to the laser point.

[0076] Exemplarily, in the process of mapping the laser points according to the determined compensation amount, for the laser points in the laser point cloud collected before the specified time, the determined compensation amount can be added on the basis of the coordinates of the laser points in the collected laser point cloud to determine the position of the laser point at the specified time; for the laser points in the laser point cloud collected after the specified time, the determined compensation amount can be subtracted on the basis of the coordinates of the laser point in the collected laser point cloud to determine the position of the laser point at the specified time.

[0077] In some embodiments, the above-mentioned marking of the laser point cloud at a specified time may include: For any laser point in the laser point cloud at a specified time, determine the RGB information of the laser point according to the pixel point corresponding to the laser point; Visualize the laser point cloud at the specified time based on the RGB information of each laser point in the laser point cloud at the specified time; The laser point cloud at a specified time is labeled based on the visual display; wherein, for any labeled target, the labeling information corresponding to the target includes importance information.

[0078] For example, considering that point cloud imaging may be ambiguous, for example, point clouds may be ambiguous due to uneven density, occlusion problems, noise or overlap, in order to improve the accuracy of point cloud annotation, in the process of visualizing the point cloud, the visual effect can be visualized according to the mapped RGB information, thereby reducing the interaction between the annotator and the image and improving the annotation efficiency.

[0079] Accordingly, for any laser point in the laser point cloud at a specified time, the RGB information of the laser point can be determined based on the pixel point corresponding to the laser point.

[0080] For example, for any laser point in the laser point cloud at a specified moment, the laser point cloud to which the laser point mapped by the laser point belongs (the laser point cloud in the above-mentioned N frames of laser point cloud) can be determined, and based on the conversion relationship between the laser radar coordinates corresponding to the laser point cloud and the image pixel coordinates, the pixel point corresponding to the laser point can be determined, and then the RGB information of the laser point can be determined.

[0081] Exemplarily, the laser point cloud at a specified moment can be visualized based on the RGB information of each laser point in the laser point cloud at the specified moment, that is, in the process of visualizing the laser point cloud, the laser point can be visualized according to the RGB information corresponding to the laser point.

[0082] Furthermore, annotation can be performed based on the laser point cloud at a specified time in the visual display.

[0083] Exemplarily, in the process of labeling the laser point cloud, for any labeled target, the labeling information corresponding to the target may include importance information.

[0084] Exemplarily, the above-mentioned importance information can be represented by whether it is within the driving area.

[0085] For example, for any marked object, the marking information corresponding to the object may include an envelope, a category, and importance information.

[0086] In some embodiments, the above-mentioned mapping relationship between each laser point in the N-frame laser point cloud and each laser point in the laser point cloud at the specified time, mapping the annotation result of the laser point cloud at the specified time back to the N-frame laser point cloud, and obtaining the annotation result of the N-frame laser point cloud may include: For any target marked in the laser point cloud at a specified time, based on the laser points included in the target and the mapping relationship between each laser point in the N-frame laser point cloud and each laser point in the laser point cloud at the specified time, the target is mapped back to the corresponding laser point cloud in the N-frame laser point cloud, and the marking result corresponding to the target in the laser point cloud of the frame is obtained; The annotation results in the N frames of laser point cloud are fine-tuned to obtain the fine-tuned annotation results.

[0087] Exemplarily, when target annotation of a laser point cloud at a specified time is implemented in the manner described in the above embodiments, for any target annotated in the laser point cloud at the specified time, the target can be mapped back to the corresponding laser point cloud in N frames of laser point cloud based on the laser points included in the target and the mapping relationship between each laser point in the N frames of laser point cloud and each laser point in the laser point cloud at the specified time, so as to obtain the annotation result corresponding to the target in the frame of laser point cloud.

[0088] For example, suppose the i-th laser point ci_T in the laser point cloud at time T (a frame of the laser point cloud in the above N frames of laser point cloud) is mapped to the laser point cloud at time T-1 (i.e., the laser point cloud at the specified time). On this basis, the laser point ci_T is annotated with the 3D target O_T-1 in the laser point cloud at time T-1 (i.e., the laser point cloud belongs to the 3D target O_T-1 in the laser point cloud at time T-1). Then a mapping relationship is established between ci_T and O_T-1, i.e., the laser point belongs to the target O_T-1. Therefore, the target O_T-1 can be mapped to time T to generate the target O_T.

[0089] Exemplarily, when target annotation results of N frames of laser point cloud are obtained, the annotation results in the N frames of laser point cloud can be fine-tuned to obtain fine-tuned annotation results to improve standard accuracy.

[0090] For example, a network form, such as pointnet, or a RANSAC algorithm is used to fine-tune the annotation results to achieve accurate annotation.

[0091] Exemplarily, for the annotation results obtained in the above manner, the annotation results can also be reviewed by methods such as time-series overlay, and abnormal targets can be modified to further improve the accuracy of the annotation.

[0092] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below with reference to specific examples.

[0093] In this embodiment, the data acquired by the vehicle-mounted laser radar and camera sensors are unified to a certain moment through multi-frame timing compensation of laser and image, and dynamic and static targets are labeled. Through the labeling post-processing link, the labeling results are mapped back to each moment, thereby improving the labeling efficiency and quality.

[0094] like Figure 2 As shown in the figure, 4 surround fisheye cameras and 1 laser radar can be configured. The cameras are installed at the front, rear, left and right of the vehicle, and the laser radar is installed on the roof.

[0095] In this embodiment, the 4D (three-dimensional space + time series) annotation implementation process can be as follows: Figure 3 As shown, each implementation process step is described below.

[0096] Camera input: a group of raw input images captured by four cameras. Each image group contains a sequence of multiple consecutive image frames, such as N frames, and each frame has a corresponding timestamp.

[0097] LiDAR input: The raw input of the LiDAR, including the x, y, z, reflectivity and other attributes of the laser point, which is synchronized with the camera in time. It also includes a multi-frame laser point cloud sequence, such as N frames, each with a corresponding timestamp.

[0098] Timing compensation: The speed of the laser point in each frame of the laser point cloud is predicted, and the RGB information of the image is assigned to the attributes of the point cloud by using the mapping relationship between the camera coordinates and the laser radar coordinates. Based on the timestamp difference of each frame and the speed information of the laser point, the position is interpolated and compensated to a certain moment (i.e. the specified moment mentioned above).

[0099] Manual labeling: The target is labeled in 3D through manual labeling. The labeled attributes include 3D target box, category, importance information, etc.

[0100] Post-annotation processing: Based on the results of manual annotation verification, the relationship between the annotation box and the point cloud is used to project it back to the original sequence image, and then automatic fine-tuning is performed.

[0101] Labeling result output: Output labeling results such as attributes and positioning information of dynamic and static targets.

[0102] Some implementation details are described below.

[0103] like Figure 4 As shown, timing compensation may include point cloud velocity prediction, camera point cloud mapping, point cloud position compensation, and point cloud aggregation. Point cloud velocity prediction: Figure 5 As shown, neural networks can be used to realize point cloud velocity prediction.

[0104] Exemplarily, the network adopts a three-channel two-stage fusion method, where: The input of the visual branch is N groups of images (each group of images can include 4 frames of images), the image height is H, the width is W, and the number of channels is C1; features can be extracted through 2D convolution / 2D sparse convolution, and then converted into BEV features (i.e., the first BEV feature mentioned above) through the perspective conversion network.

[0105] Exemplarily, the perspective conversion network can adopt LSS form or transformer form.

[0106] For example, Figure 7 As shown, the image features extracted by 2D convolution / 2D sparse convolution (i.e. Figure 7 The visual features in the figure, with a dimension of N*C1*H*W) are multiplied with the depth distribution and a maximum pooling operation is performed to obtain the first BEV feature (i.e., the visual features in the figure, with a dimension of N*C1*D*1*W).

[0107] The input of the laser branch is N frames of continuous laser frames. It is assumed that the maximum number of lasers in a single frame is P, and the number of laser channels is C2 (for example, it can include channels such as x, y, z coordinates, reflectivity, etc.).

[0108] For example, for the laser point cloud, it can be preprocessed in the form of pillar, etc., and the laser features can be extracted using 3D convolution / 3D sparse convolution to obtain the laser features to be fused.

[0109] Exemplarily, the input image and the laser point cloud may also be fused at a first level.

[0110] Among them, Figure 6 As shown in the figure, for each pixel in the input image, it has RGB three-dimensional features, which can be converted into N1-dimensional features after 2D convolution and maxpooling. The corresponding laser point has four-dimensional features of x, y, z and reflectivity. Based on the laser points in the neighborhood of the laser point (such as within a radius of d), N2-dimensional features can be obtained through MLP and maxpooling.

[0111] The input image, laser point cloud, image features corresponding to the input image (the above-mentioned N1-dimensional features), and laser features corresponding to the laser point cloud (the above-mentioned N2-dimensional features) can be concat-processed to obtain N1+N2+3+4-dimensional features, complete early fusion (i.e., first-level fusion), and obtain first-level fusion features; among them, 3 is the dimension of the input image (RGB has 3 dimensions in total), and 4 is the dimension of the laser point cloud (assuming that it includes x, y, and z coordinates, and reflectivity has 4 dimensions in total).

[0112] Exemplarily, for the first-level fusion features, features can be extracted through 2D convolution / 2D sparse convolution, and then converted into BEV features (i.e., the second BEV features mentioned above) through a perspective conversion network.

[0113] For example, Figure 7 As shown, the fused image features (i.e. Figure 7 The multidimensional features in the figure, with a dimension of N*C2*H*W) are multiplied by the depth distribution and the maximum pooling operation is performed to obtain the second BEV feature (that is, the multidimensional feature in the figure, with a dimension of N*C2*D*1*W).

[0114] Exemplarily, the first BEV feature, the laser feature to be fused, and the second BEV feature may be subjected to secondary feature fusion to obtain a final fused feature.

[0115] For example, Figure 7 As shown, on the one hand, the first BEV characteristic and the second BEV characteristic can be obtained in the above manner.

[0116] On the other hand, the image features and the fused image features are concat processed and the maxpooling operation is performed to obtain the spliced ​​fused features, and the spliced ​​fused features are multiplied with the laser features to be fused to obtain the laser fused features.

[0117] Furthermore, the cross-attention mechanism can be used to perform secondary feature fusion on the laser fusion feature, the first BEV feature, and the second BEV feature to obtain the final fusion feature.

[0118] Among them, in the process of using the cross-attention mechanism for secondary feature fusion, the correspondence between the laser fusion feature, the first BEV feature, and the second BEV feature and the key, value, and query is not limited.

[0119] For example, when the final fusion features are obtained, the speed of each pillar in the laser point cloud in the x, y, and z directions can be determined by speed prediction, and the speed of the pillar can be assigned to each laser point in the pillar.

[0120] Camera point cloud mapping: Coordinate transformation can be calculated through camera intrinsic parameters and external parameters of joint calibration of camera and lidar.

[0121] Assuming that the laser radar coordinates are (x, y, z) and the image coordinate system pixel coordinates are (u, v), the coordinate conversion relationship between them is:

[0122] Among them, s is the scale factor, M is the camera intrinsic parameter, and N is the extrinsic parameter for the joint calibration of lidar and camera.

[0123] Through the above formula, the RGB pixel information of the camera can be assigned to the corresponding laser point cloud as an important attribute.

[0124] Point cloud position compensation: Compensate N consecutive frames of laser point cloud to a fixed frame number (usually the middle moment can be selected).

[0125] For example, assume that the i-th laser point of the laser point cloud collected at time T is ci_T, and its velocities are ci_T_vx, ci_T_vy, and ci_T_vz. The specified time is time T-1, and the time difference between time T and time T-1 is t. Then the compensation amount corresponding to the laser point is: X = ci_T_vx * t Y = ci_T_vy * t Z = ci_T_vz * t And so on, position compensation is performed on all laser points in the sequence.

[0126] Point cloud aggregation: Based on the position compensation result calculated in the previous step, all laser points are mapped to a laser point cloud with a fixed frame number. Its features are the original x, y, z, reflectivity and the mapped RGB pixels.

[0127] like Figure 8 As shown, manual annotation may include model pre-annotation, manual annotation envelope, and manual annotation attributes. Model pre-labeling: Model reasoning can be used to obtain pre-labeling results to improve labeling efficiency.

[0128] Exemplarily, commonly used 3D detection models may include centerpoint (a target detection and positioning method for point cloud data), DSVT (Deep Scattering Volume Transformer), etc.

[0129] Manually annotated envelope box: Annotate the 3D outer envelope box of the target of interest (such as motor vehicles, non-motor vehicles, pedestrians, etc.).

[0130] It should be noted that in order to resolve the ambiguity caused by point cloud imaging, during the visualization of the point cloud, the visual effect can be visualized according to the mapped RGB information, reducing the interaction between the annotator and the image, thereby improving the annotation efficiency.

[0131] Manually labeled attributes: In addition to the outer envelope, the category and importance (whether it is within the drivable area, etc.) information will also be labeled.

[0132] like Fig. 9 As shown, the annotation post-processing may include point cloud annotation box mapping, position fine-tuning, and detection and verification. Point cloud annotation box mapping: Assume that the i-th laser point ci_T in the laser point cloud at time T is mapped to the laser point cloud at time T-1. On this basis, the laser point ci_T is annotated with the 3D target O_T-1 in the laser point cloud at time T-1. Then a mapping relationship is established between ci_T and O_T-1, that is, the laser point belongs to the target O_T-1. Therefore, the target O_T-1 can be mapped to time T to generate the target O_T.

[0133] Position fine-tuning: Fine-tune the annotation results obtained by mapping the point cloud annotation box.

[0134] Exemplarily, a network such as pointnet or a RANSAC algorithm can be used to fine-tune the position of the 3D box according to the initial mapping box O_T and the corresponding point cloud to achieve accurate labeling.

[0135] Detection and review: For the annotation results obtained in the above manner, you can also review the annotation results through methods such as time-series overlay, modify abnormal targets, and further improve the accuracy of annotation.

[0136] The method provided by the present application is described above. The device provided by the present application is described below: See also Fig.10 , is a schematic diagram of a structure of a marking device provided in an embodiment of the present application, such as Fig.10 As shown, the marking device may include: The acquisition unit 1010 is used to acquire N groups of images acquired by the target camera and N frames of laser point clouds acquired by the laser radar; wherein the target camera includes K vehicle-mounted cameras, the total field of view of the K vehicle-mounted cameras covers the four sides of the vehicle body, and the scanning range of the laser radar covers the four sides of the vehicle body; N≥2, K≥1, a group of images includes K frames of images acquired synchronously, the N groups of images correspond one-to-one to the N frames of laser point clouds, and the corresponding images are consistent with the acquisition time of the laser point clouds; A feature fusion unit 1020 is used to perform feature fusion on the N groups of images and the N frames of laser point clouds; A speed prediction unit 1030 is used to predict the speed of the laser points in the N frames of laser point cloud according to the fusion features to obtain the speed of each laser point in the N frames of laser point cloud; A mapping unit 1040 is used to map each laser point in the N frames of laser point cloud to a frame of laser point cloud at a specified time according to the speed of each laser point in the N frames of laser point cloud and the acquisition time of each frame of laser point cloud, so as to obtain the laser point cloud at the specified time; A labeling unit 1050 is used to label the laser point cloud at the specified time to obtain a labeling result of the laser point cloud at the specified time; The mapping unit 1040 is further used to map the annotation result of the laser point cloud at the specified moment back to the N frames of laser point cloud according to the mapping relationship between each laser point in the N frames of laser point cloud and each laser point in the laser point cloud at the specified moment, so as to obtain the annotation result of the N frames of laser point cloud.

[0137] In some embodiments, the feature fusion unit 1020 performs feature fusion on the N groups of images and the N frames of laser point clouds, including: Performing 2D feature extraction on the N groups of images, performing perspective conversion on the extracted image features to obtain a first BEV feature; performing 3D feature extraction on the N frames of laser point clouds to obtain a laser feature; And, based on the conversion relationship between the image pixel coordinates and the laser radar coordinates, the N groups of images are first-class fused with the N frames of laser point cloud to obtain first-class fusion features, 2D features are extracted from the first-class fusion features, and the extracted fusion image features are subjected to perspective conversion to obtain second BEV features; The first BEV feature, the laser feature, and the second BEV feature are subjected to secondary feature fusion to obtain a final fused feature.

[0138] In some embodiments, the feature fusion unit 1020 performs a first-level fusion on the N groups of images and the N frames of laser point clouds based on the conversion relationship between the image pixel coordinates and the laser radar coordinates to obtain a first-level fusion feature, including: For any pixel point in any image in the N groups of images, based on a conversion relationship between the image pixel coordinates of the image and the laser radar coordinates, determine a laser point corresponding to the pixel point in the laser point cloud corresponding to the image; Performing 2D feature extraction on the pixel point to obtain a feature corresponding to the pixel point; and performing feature extraction on the laser point to obtain a feature corresponding to the laser point; The pixel point, the laser point, the feature corresponding to the pixel point, and the feature corresponding to the laser point are spliced ​​to obtain the corresponding first-level fusion feature.

[0139] In some embodiments, the feature fusion unit 1020 performs secondary feature fusion on the first BEV feature, the laser feature, and the second BEV feature to obtain a final fusion feature, including: Performing feature splicing on the image feature and the fused image feature, and performing a maximum pooling operation to obtain a spliced ​​fused feature, and performing a dot product operation on the spliced ​​fused feature and the laser feature to obtain a laser fused feature; The laser fusion feature, the first BEV feature, and the second BEV feature are subjected to secondary feature fusion by using a cross attention mechanism to obtain a final fusion feature.

[0140] In some embodiments, the marking device further comprises: A laser processing unit, used for performing pillar processing on the N frames of laser point clouds to obtain processed N frames of laser point clouds; The speed prediction unit 1030 predicts the speed of the laser points in the N frames of laser point cloud according to the fusion features to obtain the speed of each laser point in the N frames of laser point cloud, including: Predicting the speed of each pillar in the N frames of laser point cloud according to the fusion features to obtain the speed of each pillar in the N frames of laser point cloud; For any pillar, the velocity of the pillar is assigned to each laser point in the pillar.

[0141] In some embodiments, the mapping unit 1040 maps each laser point in the N frames of laser point cloud to a frame of laser point cloud at a specified time according to the speed of each laser point in the N frames of laser point cloud and the acquisition time of each frame of laser point cloud to obtain the laser point cloud at the specified time, including: For any laser point in any frame of laser point cloud, based on the speed of the laser point and the time difference between the acquisition time of the frame of laser point cloud and the specified time, determine the compensation amount corresponding to the laser point; According to the compensation amount corresponding to the laser point, the laser point is mapped to the laser point cloud at a specified time.

[0142] In some embodiments, the labeling unit 1050 labels the laser point cloud at the specified time, including: For any laser point in the laser point cloud at the specified time, determine the RGB information of the laser point according to the pixel point corresponding to the laser point; Visually displaying the laser point cloud at the specified moment according to the RGB information of each laser point in the laser point cloud at the specified moment; Annotation is performed based on the visually displayed laser point cloud at the designated moment; wherein, for any annotated target, the annotation information corresponding to the target includes importance information.

[0143] In some embodiments, the mapping unit 1040 maps the annotation result of the laser point cloud at the specified time back to the N frames of laser point cloud according to the mapping relationship between each laser point in the N frames of laser point cloud and each laser point in the laser point cloud at the specified time, and obtains the annotation result of the N frames of laser point cloud, including: For any target marked in the laser point cloud at the specified time, based on the laser points included in the target and the mapping relationship between each laser point in the N frames of laser point cloud and each laser point in the laser point cloud at the specified time, the target is mapped back to the corresponding laser point cloud in the N frames of laser point cloud to obtain a marking result corresponding to the target in the frame of laser point cloud; The annotation results in the N frames of laser point cloud are fine-tuned to obtain a fine-tuned annotation result.

[0144] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the marking method described above.

[0145] See also Fig.11 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 1101 and a memory 1102 storing machine executable instructions. The processor 1101 and the memory 1102 may communicate via a system bus 1103. Furthermore, by reading and executing the machine executable instructions corresponding to the annotation logic in the memory 1102, the processor 1101 may execute the annotation method described above.

[0146] The memory 1102 mentioned herein may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (RadomAccess Memory, random access memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0147] In some embodiments, a machine-readable storage medium is also provided. Fig.11 The memory 1102 in the machine readable storage medium stores machine executable instructions, and the machine executable instructions implement the above-described marking method when executed by the processor. For example, the storage medium can be ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

Claims

1. A marking method, characterized in that: include: Obtain N groups of images collected by the target camera and N frames of laser point clouds collected by the laser radar; wherein the target camera includes K vehicle-mounted cameras, the total field of view of the K vehicle-mounted cameras covers the four sides of the vehicle body, and the scanning range of the laser radar covers the four sides of the vehicle body; N≥2, K≥1, a group of images includes K frames of images collected synchronously, the N groups of images correspond one-to-one to the N frames of laser point clouds, and the corresponding images are consistent with the acquisition time of the laser point clouds; Performing feature fusion on the N groups of images and the N frames of laser point clouds, and predicting the speed of the laser points in the N frames of laser point clouds according to the fusion features to obtain the speed of each laser point in the N frames of laser point clouds; According to the speed of each laser point in the N frames of laser point cloud and the acquisition time of each frame of laser point cloud, each laser point in the N frames of laser point cloud is mapped to a frame of laser point cloud at a specified time to obtain the laser point cloud at the specified time; Annotating the laser point cloud at the specified time to obtain an annotation result of the laser point cloud at the specified time; According to the mapping relationship between each laser point in the N frames of laser point cloud and each laser point in the laser point cloud at the specified time, the annotation result of the laser point cloud at the specified time is mapped back to the N frames of laser point cloud to obtain the annotation result of the N frames of laser point cloud.

2. The method according to claim 1, characterized in that The performing feature fusion on the N groups of images and the N frames of laser point clouds includes: Performing 2D feature extraction on the N groups of images, performing perspective conversion on the extracted image features to obtain a first BEV feature; performing 3D feature extraction on the N frames of laser point clouds to obtain a laser feature; And, based on the conversion relationship between the image pixel coordinates and the laser radar coordinates, the N groups of images are first-class fused with the N frames of laser point cloud to obtain first-class fusion features, 2D features are extracted from the first-class fusion features, and the extracted fusion image features are subjected to perspective conversion to obtain second BEV features; The first BEV feature, the laser feature, and the second BEV feature are subjected to secondary feature fusion to obtain a final fused feature.

3. The method according to claim 2, characterized in that Based on the conversion relationship between the image pixel coordinates and the laser radar coordinates, the N groups of images and the N frames of laser point clouds are first-level fused to obtain first-level fusion features, including: For any pixel point in any image in the N groups of images, based on a conversion relationship between the image pixel coordinates of the image and the laser radar coordinates, determine a laser point corresponding to the pixel point in the laser point cloud corresponding to the image; Performing 2D feature extraction on the pixel point to obtain a feature corresponding to the pixel point; and performing feature extraction on the laser point to obtain a feature corresponding to the laser point; The pixel point, the laser point, the feature corresponding to the pixel point, and the feature corresponding to the laser point are spliced ​​to obtain the corresponding first-level fusion feature.

4. The method according to claim 2, characterized in that: The performing secondary feature fusion on the first BEV feature, the laser feature, and the second BEV feature to obtain a final fusion feature includes: Performing feature splicing on the image feature and the fused image feature, and performing a maximum pooling operation to obtain a spliced ​​fused feature, and performing a dot product operation on the spliced ​​fused feature and the laser feature to obtain a laser fused feature; The laser fusion feature, the first BEV feature, and the second BEV feature are subjected to secondary feature fusion by using a cross attention mechanism to obtain a final fusion feature.

5. The method according to claim 1, characterized in that The method further comprises: Performing pillar processing on the N frames of laser point clouds to obtain processed N frames of laser point clouds; The method of predicting the speed of the laser points in the N frames of laser point cloud according to the fusion features to obtain the speed of each laser point in the N frames of laser point cloud includes: Predicting the speed of each pillar in the N frames of laser point cloud according to the fusion features to obtain the speed of each pillar in the N frames of laser point cloud; For any pillar, the velocity of the pillar is assigned to each laser point in the pillar.

6. The method according to claim 1, characterized in that The method maps each laser point in the N frames of laser point cloud to a frame of laser point cloud at a specified time according to the speed of each laser point in the N frames of laser point cloud and the acquisition time of each frame of laser point cloud to obtain the laser point cloud at the specified time, including: For any laser point in any frame of laser point cloud, based on the speed of the laser point and the time difference between the acquisition time of the frame of laser point cloud and the specified time, determine the compensation amount corresponding to the laser point; According to the compensation amount corresponding to the laser point, the laser point is mapped to the laser point cloud at a specified time.

7. The method according to claim 1, characterized in that The marking of the laser point cloud at the specified time includes: For any laser point in the laser point cloud at the specified time, determine the RGB information of the laser point according to the pixel point corresponding to the laser point; Visually displaying the laser point cloud at the specified moment according to the RGB information of each laser point in the laser point cloud at the specified moment; Annotation is performed based on the visually displayed laser point cloud at the designated moment; wherein, for any annotated target, the annotation information corresponding to the target includes importance information.

8. The method according to claim 1, characterized in that According to the mapping relationship between each laser point in the N frames of laser point cloud and each laser point in the laser point cloud at the specified time, mapping the annotation result of the laser point cloud at the specified time back to the N frames of laser point cloud to obtain the annotation result of the N frames of laser point cloud, including: For any target marked in the laser point cloud at the specified time, based on the laser points included in the target and the mapping relationship between each laser point in the N frames of laser point cloud and each laser point in the laser point cloud at the specified time, the target is mapped back to the corresponding laser point cloud in the N frames of laser point cloud to obtain a marking result corresponding to the target in the frame of laser point cloud; The annotation results in the N frames of laser point cloud are fine-tuned to obtain a fine-tuned annotation result.

9. A marking device, characterized in that: include: An acquisition unit is used to acquire N groups of images acquired by a target camera and N frames of laser point clouds acquired by a laser radar; wherein the target camera includes K vehicle-mounted cameras, the total field of view of the K vehicle-mounted cameras covers the four sides of the vehicle body, and the scanning range of the laser radar covers the four sides of the vehicle body; N≥2, K≥1, a group of images includes K frames of images acquired synchronously, the N groups of images correspond one-to-one to the N frames of laser point clouds, and the corresponding images are consistent with the acquisition time of the laser point clouds; A feature fusion unit, used for performing feature fusion on the N groups of images and the N frames of laser point clouds; A speed prediction unit, used for predicting the speed of the laser points in the N frames of laser point cloud according to the fusion features, to obtain the speed of each laser point in the N frames of laser point cloud; A mapping unit, configured to map each laser point in the N frames of laser point cloud to a frame of laser point cloud at a specified time according to the speed of each laser point in the N frames of laser point cloud and the acquisition time of each frame of laser point cloud, so as to obtain the laser point cloud at the specified time; A labeling unit, used to label the laser point cloud at the specified time to obtain a labeling result of the laser point cloud at the specified time; The mapping unit is further used to map the annotation result of the laser point cloud at the specified moment back to the N frames of laser point cloud according to the mapping relationship between each laser point in the N frames of laser point cloud and each laser point in the laser point cloud at the specified moment, so as to obtain the annotation result of the N frames of laser point cloud.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method according to any one of claims 1 to 8.

11. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images