Bird-eye view BEV data labeling method and device and related product

Abstract

The invention discloses an aerial view BEV data labeling method and device and a related product. Obtaining a non-ground point cloud in the three-dimensional point cloud information based on the three-dimensional point cloud information; and based on the plurality of different environment look-around images, obtaining corresponding obstacle information in the plurality of different environment look-around images; performing matching processing on the non-ground point clouds based on corresponding obstacle information in a plurality of different environment look-around images to obtain non-ground point clouds with categories; performing screening processing on the non-ground point cloud based on a vehicle-mounted laser radar to obtain a target point cloud point closest to the vehicle-mounted laser radar in the non-ground point cloud; and based on the non-ground point cloud, the non-ground point cloud with the category and the target point cloud, labeling and generating a labeling result of the BEV data. In this way, the annotation efficiency and the annotation accuracy of the BEV data are improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving perception technology, and in particular to a method, apparatus and related products for annotating bird's-eye view BEV data. Background Technology

[0002] In the field of autonomous driving technology, the construction of deep learning models heavily relies on accurately labeled bird's-eye view BEV data. However, current related technical solutions mainly rely on manual methods to label BEV data, which not only significantly reduces labeling efficiency but also leads to a substantial decrease in labeling accuracy. Therefore, how to improve the labeling efficiency and accuracy of BEV data is a key issue of concern to those skilled in the art. Summary of the Invention

[0003] To address the aforementioned issues, this application provides a method, apparatus, and related products for labeling BEV data from bird's-eye views, in order to improve the efficiency and accuracy of BEV data labeling.

[0004] The embodiments of this application disclose the following technical solutions:

[0005] The first aspect of this application provides a method for annotating BEV data in a bird's-eye view. This method for annotating BEV data in a bird's-eye view includes:

[0006] The data acquisition vehicle obtains 3D point cloud information of the parking environment collected by its onboard LiDAR and environmental surround view images of the parking environment collected by onboard surround view cameras at different locations, wherein the 3D point cloud information and multiple different environmental surround view images are at the same timestamp.

[0007] Based on the three-dimensional point cloud information, obtain non-ground point cloud information in the three-dimensional point cloud information; and based on the multiple different environmental surround view images, obtain obstacle information corresponding to each of the multiple different environmental surround view images.

[0008] Based on the obstacle information corresponding to the multiple different environmental surround view images, the non-ground point cloud is matched to obtain a non-ground point cloud with categories.

[0009] Based on the vehicle-mounted lidar, the non-ground point cloud is filtered to obtain the target point cloud point that is closest to the vehicle-mounted lidar in the non-ground point cloud, wherein the target point cloud point indicates a ground boundary point.

[0010] Based on the non-ground point cloud, the non-ground point cloud with categories, and the target point cloud points, the annotation results of BEV data are generated.

[0011] In one feasible implementation, before performing matching processing on the non-ground point cloud based on obstacle information corresponding to the plurality of different environmental surround view images to obtain a categorized non-ground point cloud, the method further includes:

[0012] Multiple coordinate transformation rules are obtained, one of which corresponds to the transformation rule between the vehicle-mounted LiDAR and a vehicle-mounted surround-view camera;

[0013] The obstacle information includes obstacles corresponding to a first category and obstacles corresponding to a second category, wherein the first category focuses on the obstacle as a whole, and the second category focuses on the obstacle boundary; the matching process of the non-ground point cloud based on the obstacle information corresponding to the multiple different environmental surround view images to obtain a non-ground point cloud with categories includes:

[0014] For each environmental panoramic image, based on the coordinate transformation rules corresponding to the environmental panoramic image, the obstacles in the environmental panoramic image corresponding to the first category are matched with the point cloud in the non-ground point cloud corresponding to the first category to obtain the non-ground point cloud corresponding to the first category.

[0015] For each environmental panoramic image, based on the coordinate transformation rule corresponding to the environmental panoramic image, the obstacles in the environmental panoramic image corresponding to the second category are matched with the non-ground point cloud to obtain the non-ground point cloud corresponding to the second category;

[0016] Based on the non-ground point cloud corresponding to the first category and the non-ground point cloud corresponding to the second category, a non-ground point cloud with categories is obtained.

[0017] In one feasible implementation, the step of matching the obstacles in the environmental surround view image corresponding to the second category with the non-ground point cloud to obtain the non-ground point cloud corresponding to the second category includes:

[0018] The Z-value of the non-ground point cloud is adjusted to obtain a target Z-value of the non-ground point cloud, wherein the target Z-value is greater than the Z-value.

[0019] The obstacles in the environmental panoramic image corresponding to the second category are matched with the non-ground point cloud containing the target Z value to obtain the non-ground point cloud corresponding to the second category.

[0020] In one feasible implementation, the step of filtering the non-ground point cloud based on the vehicle-mounted LiDAR to obtain the target point cloud point closest to the vehicle-mounted LiDAR in the non-ground point cloud includes:

[0021] Using the vehicle-mounted lidar as the center, a 360-degree angle range is determined;

[0022] For each angle within the 360-degree angle range, the non-ground point cloud at that angle is filtered to obtain the target point cloud point closest to the vehicle-mounted LiDAR in the non-ground point cloud at that angle.

[0023] In one feasible implementation, the annotation result for generating BEV data based on the non-ground point cloud, the categorized non-ground point cloud, and the target point cloud points includes:

[0024] The non-ground point cloud is rasterized to obtain the first BEV;

[0025] The non-ground point cloud with categories is rasterized to obtain the second BEV;

[0026] The target point cloud points are rasterized to obtain the third BEV;

[0027] The second BEV and the third BEV are used as the labeling results of the first BEV.

[0028] In one feasible implementation, obtaining the non-ground point cloud from the three-dimensional point cloud information based on the three-dimensional point cloud information includes:

[0029] The 3D point cloud information is classified based on a point cloud classification algorithm to obtain ground point cloud and non-ground point cloud in the 3D point cloud information;

[0030] Delete the ground point cloud from the three-dimensional point cloud information, and retain the non-ground point cloud from the three-dimensional point cloud information.

[0031] In one feasible implementation, obtaining obstacle information corresponding to each of the multiple different environmental surround view images includes:

[0032] For each environmental surround view image, the environmental surround view image is segmented using an image segmentation model to obtain obstacle information in the environmental surround view image.

[0033] A second aspect of this application provides a device for annotating bird's-eye view BEV data. This device for annotating bird's-eye view BEV data includes:

[0034] The image information acquisition unit is used to acquire the three-dimensional point cloud information of the parking environment collected by the vehicle-mounted lidar of the data acquisition vehicle, and the environmental surround view images of the parking environment collected by the vehicle-mounted surround view cameras at different locations, wherein the three-dimensional point cloud information and multiple different environmental surround view images are at the same timestamp.

[0035] The point cloud information acquisition unit is used to obtain non-ground point clouds in the three-dimensional point cloud information based on the three-dimensional point cloud information; and to obtain obstacle information corresponding to each of the multiple different environmental surround view images based on the multiple different environmental surround view images.

[0036] The category point cloud acquisition unit is used to match the non-ground point cloud based on the obstacle information corresponding to the multiple different environmental surround view images to obtain a category-based non-ground point cloud;

[0037] The target point cloud acquisition unit is used to filter the non-ground point cloud based on the vehicle-mounted lidar to obtain the target point cloud point that is closest to the vehicle-mounted lidar in the non-ground point cloud, wherein the target point cloud point indicates a ground boundary point.

[0038] The annotation result generation unit is used to generate annotation results for BEV data based on the non-ground point cloud, the non-ground point cloud with categories, and the target point cloud points.

[0039] In one feasible implementation, the device further includes:

[0040] The transformation rule acquisition unit is used to acquire multiple coordinate transformation rules, one of which corresponds to the transformation rule between the vehicle-mounted LiDAR and a vehicle-mounted surround view camera.

[0041] The obstacle information includes obstacles corresponding to a first category and obstacles corresponding to a second category, wherein the first category focuses on the obstacle as a whole, and the second category focuses on the obstacle boundary; the category point cloud acquisition unit includes:

[0042] The first category point cloud acquisition unit is used to, for each environmental panoramic image, based on the coordinate transformation rules corresponding to the environmental panoramic image, match the obstacles in the environmental panoramic image corresponding to the first category with the point cloud in the non-ground point cloud corresponding to the first category to obtain the non-ground point cloud corresponding to the first category.

[0043] The second category point cloud acquisition unit is used to, for each environmental panoramic image, based on the coordinate transformation rule corresponding to the environmental panoramic image, match the obstacles in the environmental panoramic image corresponding to the second category with the non-ground point cloud to obtain the non-ground point cloud corresponding to the second category.

[0044] The category-based point cloud acquisition unit is used to obtain a category-based non-ground point cloud based on the non-ground point cloud corresponding to the first category and the non-ground point cloud corresponding to the second category.

[0045] In one feasible implementation, the second type of point cloud acquisition unit is specifically used for:

[0046] The Z-value of the non-ground point cloud is adjusted to obtain a target Z-value of the non-ground point cloud, wherein the target Z-value is greater than the Z-value.

[0047] The obstacles in the environmental panoramic image corresponding to the second category are matched with the non-ground point cloud containing the target Z value to obtain the non-ground point cloud corresponding to the second category.

[0048] In one feasible implementation, the target point cloud point acquisition unit is specifically used for:

[0049] Using the vehicle-mounted lidar as the center, a 360-degree angle range is determined;

[0050] For each angle within the 360-degree angle range, the non-ground point cloud at that angle is filtered to obtain the target point cloud point closest to the vehicle-mounted LiDAR in the non-ground point cloud at that angle.

[0051] In one feasible implementation, the annotation result generation unit is specifically used for:

[0052] The non-ground point cloud is rasterized to obtain the first BEV;

[0053] The non-ground point cloud with categories is rasterized to obtain the second BEV;

[0054] The target point cloud points are rasterized to obtain the third BEV;

[0055] The second BEV and the third BEV are used as the labeling results of the first BEV.

[0056] In one feasible implementation, the point cloud information acquisition unit is specifically used for:

[0057] The 3D point cloud information is classified based on a point cloud classification algorithm to obtain ground point cloud and non-ground point cloud in the 3D point cloud information;

[0058] Delete the ground point cloud from the three-dimensional point cloud information, and retain the non-ground point cloud from the three-dimensional point cloud information.

[0059] In one feasible implementation, the point cloud information acquisition unit is specifically used for:

[0060] For each environmental surround view image, the environmental surround view image is segmented using an image segmentation model to obtain obstacle information in the environmental surround view image.

[0061] A third aspect of this application provides a computer device. The computer device includes:

[0062] A memory on which computer programs are stored;

[0063] A processor for executing the computer program in the memory to implement the steps of the method for annotating bird's-eye view BEV data provided in the first aspect.

[0064] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the program implements the steps of the annotation method for bird's-eye view BEV data provided in the first aspect.

[0065] Compared with the prior art, this application has the following beneficial effects:

[0066] This application first acquires 3D point cloud information of the parking environment collected by the vehicle-mounted LiDAR of a data acquisition vehicle, and environmental surround view images of the parking environment collected by vehicle-mounted surround view cameras at different locations. Next, based on the 3D point cloud information, non-ground point clouds can be obtained. Furthermore, based on multiple different environmental surround view images, obstacle information corresponding to each of the multiple environmental surround view images can be obtained. Then, based on the obstacle information corresponding to each of the multiple different environmental surround view images, the non-ground point clouds can be matched to obtain categorized non-ground point clouds. Finally, based on the vehicle-mounted LiDAR, the non-ground point clouds can be filtered to obtain the target point cloud points in the non-ground point clouds that are closest to the vehicle-mounted LiDAR. Finally, based on the non-ground point clouds, the categorized non-ground point clouds, and the target point cloud points, annotation results for BEV data can be generated. The 3D point cloud information and multiple different environmental surround view images are at the same timestamp, and the target point cloud points indicate ground boundary points.

[0067] As can be seen, this application first obtains non-ground point clouds from 3D point cloud information at the same timestamp, and corresponding obstacle information from multiple different environmental surround view images. Then, the non-ground point clouds are matched with the obstacle information to obtain categorized non-ground point clouds, and ground boundary points within the non-ground point clouds can be determined, thus achieving automatic annotation of BEV data. Therefore, compared to related technical solutions, this application achieves automated annotation (annotating categorized point clouds and ground boundary points within the point clouds), reducing reliance on manual labor, quickly generating accurate and reliable annotation data, and improving the annotation efficiency and accuracy of BEV data. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 A flowchart illustrating a method for annotating BEV data in a bird's-eye view, as provided in an embodiment of this application;

[0070] Figure 2 This is a schematic diagram of three-dimensional point cloud information in a bird's-eye view BEV data annotation method provided in an embodiment of this application;

[0071] Figure 3 A schematic diagram of the environmental surround view image in a method for annotating BEV data from a bird's-eye view provided in an embodiment of this application;

[0072] Figure 4 A schematic diagram of BEV in a bird's-eye view BEV data annotation method provided in an embodiment of this application;

[0073] Figure 5 This application provides a flowchart of the entire process of data annotation in a method for annotating BEV data in a bird's-eye view.

[0074] Figure 6 This is a schematic diagram of a bird's-eye view BEV data annotation device provided in an embodiment of this application. Detailed Implementation

[0075] As described earlier, in the field of autonomous driving technology, the construction of deep learning models heavily relies on accurately labeled bird's-eye view BEV data. However, current related technical solutions mainly rely on manual methods for BEV data labeling, which has many drawbacks. On the one hand, manual labeling makes it difficult to guarantee the accuracy, completeness, and consistency of the labeled data, leading to a significant decrease in labeling accuracy. The labeling results are greatly affected by subjective factors, potentially resulting in labeling errors or inconsistent standards, and also reducing labeling efficiency, thereby affecting the training effect and performance of the model. On the other hand, manual labeling requires a large investment of human resources and consumes a lot of time, resulting in high labeling costs. Therefore, how to improve the labeling efficiency and accuracy of BEV data is a key issue of concern to those skilled in the art.

[0076] To address the above issues, this application provides a solution, proposing a method, apparatus, and related products for annotating bird's-eye view BEV data, aiming to improve the efficiency and accuracy of BEV data annotation. The technical solution of this application first acquires 3D point cloud information of the parking environment collected by the vehicle-mounted LiDAR of a data acquisition vehicle, and environmental surround view images of the parking environment collected by vehicle-mounted surround view cameras at different locations. Next, based on the 3D point cloud information, non-ground point clouds can be obtained from the 3D point cloud information. Furthermore, based on multiple different environmental surround view images, obstacle information corresponding to each of the multiple environmental surround view images can be obtained. Then, based on the obstacle information corresponding to each of the multiple different environmental surround view images, the non-ground point clouds can be matched to obtain non-ground point clouds with categories. Additionally, based on the vehicle-mounted LiDAR, the non-ground point clouds can be filtered to obtain the target point cloud points in the non-ground point clouds that are closest to the vehicle-mounted LiDAR. Finally, based on the non-ground point clouds, the non-ground point clouds with categories, and the target point cloud points, the annotation results of the BEV data can be generated. The 3D point cloud information and multiple different environmental panoramic images are at the same timestamp, and the target point cloud points indicate the ground boundary points.

[0077] As can be seen, this application first obtains non-ground point clouds from 3D point cloud information at the same time stamp, and obstacle information corresponding to multiple different environmental surround view images. Then, the non-ground point clouds are matched with the obstacle information to obtain categorized non-ground point clouds, and ground boundary points within the non-ground point clouds can be determined, thus achieving automatic annotation of BEV data. Compared to related technical solutions, this application achieves automated annotation (annotating categorized point clouds and ground boundary points within the point clouds), reducing reliance on manual labor, quickly generating accurate and reliable annotation data, improving the annotation efficiency and accuracy of BEV data, thereby improving the annotation quality of BEV data and reducing the annotation cost. This ultimately meets the demand for large amounts of high-quality annotation data in the autonomous driving field, promotes the development and application of BEV models, and enhances the safety and reliability of autonomous driving technology.

[0078] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0079] See Figure 1 This figure is a flowchart of a method for annotating BEV data in a bird's-eye view, provided in an embodiment of this application. Figure 1 The annotation methods for the BEV data in the bird's-eye view shown include:

[0080] S101: Acquire the 3D point cloud information of the parking environment collected by the vehicle-mounted LiDAR of the data acquisition vehicle, and the environmental surround view images of the parking environment collected by the vehicle-mounted surround view cameras at different locations.

[0081] In this step, the 3D point cloud information and multiple different environmental surround view images are at the same timestamp. For example, the timestamp for obtaining the 3D point cloud information is timestamp A, the timestamp for obtaining environmental surround view image a is timestamp A, and the timestamp for obtaining environmental surround view image b is timestamp A. Understandably, in this application, an onboard LiDAR and four onboard surround view cameras located at different positions can be mounted on the data acquisition vehicle to collect environmental information (i.e., 3D point cloud information and environmental surround view images) of the parking environment.

[0082] The vehicle-mounted LiDAR includes a target LiDAR and a blind spot LiDAR. The target LiDAR is used to collect 3D point cloud information of the parking environment, while the blind spot LiDAR is used to collect point cloud information of the blind spots in the parking environment, effectively solving the near-field blind spot problem of the target LiDAR. Figure 2 As shown, Figure 2 This is a schematic diagram of 3D point cloud information in a bird's-eye view BEV data annotation method provided in an embodiment of this application. Figure 2 The image shows 3D point cloud information of the parking environment collected by the vehicle-mounted lidar.

[0083] This vehicle-mounted surround-view camera is used to capture environmental surround-view images of the parking scene from different angles to ensure comprehensive coverage of all areas of the parking scene. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of the environmental surround view image in a bird's-eye view BEV data annotation method provided in an embodiment of this application. Figure 3 Images (a), (b), (c), and (d) in the figure show environmental surround view images of the parking environment captured by vehicle-mounted surround view cameras at different locations.

[0084] It should be noted that the target LiDAR can be located at the front of the data acquisition vehicle, the blind spot LiDAR can be located at the rear of the data acquisition vehicle, and the four vehicle-mounted surround view cameras in different positions can be located at the front, rear, left, and right ends of the data acquisition vehicle, respectively. The positions of the target LiDAR, blind spot LiDAR, and vehicle-mounted surround view cameras can also be set according to actual needs in practical applications.

[0085] It should also be noted that this application allows for time synchronization settings for the target LiDAR, blind spot LiDAR, and multiple different vehicle-mounted surround view cameras. This ensures that the 3D point cloud information and multiple different environmental surround view images are at the same timestamp, making the data correspond in the time dimension and facilitating subsequent data processing and analysis. Understandably, this application first acquires all 3D point cloud information and all environmental surround view images of the parking environment based on the data acquisition vehicle, and then acquires the 3D point cloud information and environmental surround view images of the parking environment at the same timestamp.

[0086] S102: Based on the three-dimensional point cloud information, obtain the non-ground point cloud in the three-dimensional point cloud information; and based on the multiple different environmental surround view images, obtain the obstacle information corresponding to each of the multiple different environmental surround view images.

[0087] Specifically, this application first classifies 3D point cloud information based on a point cloud classification algorithm to obtain ground point clouds and non-ground point clouds. Then, ground point clouds are deleted, while non-ground point clouds are retained. The point cloud classification algorithm includes open-source algorithms, and non-ground point clouds contain key information such as obstacles. Thus, this application retains only non-ground point clouds, laying the foundation for subsequent point cloud category matching and rasterization operations.

[0088] It should be noted that in this application, the 3D point cloud information can also be cropped to a required range (e.g., within 5 meters of the data acquisition vehicle) to remove unnecessary point cloud information, reduce data processing volume, and improve processing efficiency. Subsequently, the cropped 3D point cloud information is classified based on a point cloud classification algorithm to obtain the ground point cloud and non-ground point cloud within the cropped 3D point cloud information.

[0089] Furthermore, for each of the multiple different environmental surround view images, an image segmentation model is used to segment the environmental surround view image to obtain obstacle information. Specifically, this application can use an image segmentation model to segment the environmental surround view image, obtain the category of each pixel in the environmental surround view image, and thus obtain obstacle information. This lays the foundation for subsequent point cloud category matching.

[0090] The image segmentation model includes a look-around FreeSpace model, and obstacle information includes obstacles of a first category and obstacles of a second category. The first category focuses on the obstacle as a whole, while the second category focuses on the obstacle boundaries. For example, obstacles of the first category include vehicles and pedestrians, while obstacles of the second category include walls, pillars, guardrails, and curbs.

[0091] It should also be noted that the training process of this image segmentation model is as follows:

[0092] First, obtain the view-around sample image and the label category corresponding to each sample pixel in the view-around sample image. Then, use the segmentation model to be trained to predict the view-around sample image, obtaining the predicted category corresponding to each sample pixel in the view-around sample image. Finally, based on the difference between the label category and the predicted category of each sample pixel in the view-around sample image, adjust the model parameters until the adjusted model meets the model training cutoff condition, ending the training and obtaining the image segmentation model. One label category and one predicted category are used as a pair of model training data.

[0093] S103: Based on the obstacle information corresponding to the multiple different environmental surround view images, the non-ground point cloud is matched to obtain a non-ground point cloud with categories.

[0094] It should be noted that before performing the matching processing of non-ground point clouds based on obstacle information corresponding to multiple different environmental surround view images to obtain categorized non-ground point clouds, this application can also obtain multiple coordinate transformation rules. These multiple coordinate transformation rules indicate the transformation rules between the vehicle-mounted LiDAR and multiple vehicle-mounted surround view cameras, with one coordinate transformation rule corresponding to the transformation rule between the vehicle-mounted LiDAR and one vehicle-mounted surround view camera. In this way, this application determines the coordinate transformation relationship between the vehicle-mounted LiDAR and each vehicle-mounted surround view camera through a precise calibration method, thereby establishing a spatial mapping bridge between the point cloud data and the surround view image data.

[0095] Specifically, in this application, for each environmental surround view image, based on the coordinate transformation rules corresponding to that environmental surround view image, the obstacles corresponding to the first category in the environmental surround view image are matched with the point clouds in the non-ground point cloud corresponding to the first category to obtain the non-ground point cloud corresponding to the first category. It is understandable that since the image segmentation model can detect the entire obstacle, and the point cloud can better represent the entire obstacle, for obstacles corresponding to the first category (such as vehicles and pedestrians), the obstacles corresponding to the first category in the environmental surround view image and the point clouds in the non-ground point cloud corresponding to the first category can be directly projected and matched to obtain the non-ground point cloud corresponding to the first category.

[0096] Furthermore, for each environmental panoramic image, based on the coordinate transformation rule corresponding to that environmental panoramic image, the obstacles corresponding to the second category in the environmental panoramic image are matched with the non-ground point cloud to obtain the non-ground point cloud corresponding to the second category. Specifically, in this application, the Z-values ​​of the non-ground point clouds can be adjusted to obtain the target Z-values ​​of the non-ground point clouds. That is, the Z-values ​​(values ​​on the vertical axis) of all point clouds can be adjusted to obtain the target Z-values ​​of all point clouds, where the target Z-value is greater than the Z-value. Afterwards, the obstacles corresponding to the second category in the environmental panoramic image can be matched with the non-ground point cloud containing the target Z-value to obtain the non-ground point cloud corresponding to the second category.

[0097] Understandably, since image segmentation models can detect obstacle boundaries (i.e., the boundary with the drivable area), and point clouds fail to adequately represent obstacle boundaries (i.e., fewer point clouds hit by the vehicle-mounted LiDAR at ground boundaries), for drivable area boundary categories such as walls, pillars, guardrails, and curbs, the Z-values ​​of all point clouds can be adjusted to a certain range of ground height (e.g., if the ground height from the radar is 5 meters, then the Z-values ​​are adjusted to 5 meters), and then projection matching is performed. This assigns a corresponding category to each point cloud, resulting in the non-ground point cloud corresponding to the second category. Thus, this application, even when point clouds fail to adequately represent obstacle boundaries, can achieve matching for the second category of point clouds by adjusting the point cloud Z-values.

[0098] Finally, the non-ground point clouds corresponding to the first category and the second category can be identified as categorized non-ground point clouds. Thus, in this application, by using vehicle-mounted LiDAR to calibrate each vehicle-mounted surround-view camera, the non-ground point clouds are mapped onto the obstacle information of each surround-view image, thereby assigning a category to each point cloud and achieving automatic labeling to a certain extent.

[0099] S104: Based on the vehicle-mounted lidar, the non-ground point cloud is filtered to obtain the target point cloud point that is closest to the vehicle-mounted lidar in the non-ground point cloud.

[0100] Specifically, in this application, the vehicle-mounted LiDAR can be used as the center to determine a 360-degree angle range centered on the vehicle-mounted LiDAR. After that, for each angle in the 360-degree angle range, the non-ground point cloud under that angle can be filtered to obtain the target point cloud point closest to the vehicle-mounted LiDAR in the non-ground point cloud under that angle.

[0101] Understandably, in this application, the radar coordinate origin is used as the center, and within a 360-degree range, the nearest non-ground point cloud point to the center is found at each angle as the ground boundary point. The target point cloud point indicates the ground boundary point, and the area from the target point cloud point to the vehicle's interior represents the vehicle's drivable area. It should be noted that this application can also employ filtering and other methods to ignore noise, ensuring that the found ground boundary points are accurate and reliable, and can truly reflect the ground boundaries in parking scenarios.

[0102] S105: Based on the non-ground point cloud, the non-ground point cloud with categories, and the target point cloud points, generate the annotation results of BEV data.

[0103] Specifically, in this application, non-terrestrial point clouds can be rasterized to obtain a first BEV (bird's-eye view); non-terrestrial point clouds with categories can be rasterized to obtain a second BEV; target point cloud points can be rasterized to obtain a third BEV; finally, the second and third BEVs can be used as the annotation results of the first BEV. Figure 4 As shown, Figure 4 This is a schematic diagram of BEV in a bird's-eye view BEV data annotation method provided in an embodiment of this application. Figure 4 (a) in the text represents the second BEV. Figure 4 (b) in the text represents the first BEV. Figure 4 (c) in the text represents the third BEV.

[0104] In this system, the first BEV (Boundary Elevated Vehicle) can serve as a training sample for the downstream model, aiming to detect obstacles; the second and third BEVs can serve as training labels for the downstream model, with the second BEV used to identify obstacles with categories and the third BEV used to identify drivable areas. Thus, compared to traditional manual annotation, this application achieves annotation results in seconds through automated operations such as point cloud matching and coloring, improving annotation efficiency by over 100 times, significantly saving time and meeting the rapid demand for large amounts of labeled data in the autonomous driving field.

[0105] In one feasible implementation, non-ground point clouds, non-ground point clouds with categories, and target point clouds can be stored in three channels of an image to generate the final BEV ground truth (model training samples and model training labels), which can be directly used for training the BEV model.

[0106] like Figure 5 As shown, Figure 5 This is a flowchart illustrating the entire process of data annotation in a bird's-eye view BEV data annotation method provided in an embodiment of this application. Figure 5First, the system acquires 3D point cloud information of the parking environment from the vehicle-mounted LiDAR and surround-view images of the parking environment from cameras at different locations. Then, based on these images, an image segmentation model can be used to obtain obstacle information corresponding to each of the multiple surround-view images. Finally, a point cloud classification algorithm can be used to obtain non-ground and ground point clouds from the 3D point cloud information.

[0107] Next, based on obstacle information corresponding to multiple different environmental surround view images, the non-ground point cloud can be matched to obtain a non-ground point cloud with categories. Furthermore, the non-ground point cloud can be filtered based on the vehicle-mounted LiDAR to obtain the target point cloud points closest to the vehicle-mounted LiDAR. Finally, the non-ground point cloud can be rasterized into a first BEV, the non-ground point cloud with categories can be rasterized into a second BEV, and the target point cloud points can be rasterized into a third BEV. The second and third BEVs are then used as the annotation results for the first BEV. Thus, in this application, by matching the radar point cloud with the segmentation and inference results of the surround view image, labels are assigned to the point cloud points. Then, the point cloud is rasterized to generate BEV ground truth values ​​for BEV model training. This eliminates subjective errors in the manual annotation process, ensures the consistency and accuracy of the annotation results, significantly improves the reliability of the labeled data (GT), provides a high-quality data foundation for BEV model training, and helps improve the model's performance and generalization ability.

[0108] As can be seen, in this embodiment, non-ground point clouds from 3D point cloud information at the same timestamp and obstacle information corresponding to multiple different environmental surround view images can be obtained first. Then, the non-ground point clouds and obstacle information can be matched to obtain non-ground point clouds with categories, and ground boundary points in the non-ground point clouds can be determined, thereby achieving automatic annotation of BEV data. Thus, this application fully utilizes 3D point cloud information collected by LiDAR and surround view images collected by surround view cameras, fusing the two through algorithms to achieve accurate identification and annotation of objects in parking scenarios. Compared with related technical solutions, it has stronger environmental adaptability and accuracy. This achieves automated annotation (annotating point clouds with categories and ground boundary points in the point clouds), reducing reliance on manual labor, quickly generating accurate and reliable annotation data, improving the annotation efficiency and accuracy of BEV data, thereby improving the annotation quality of BEV data and reducing the annotation cost of BEV data. This further meets the demand for large amounts of high-quality annotation data in the field of autonomous driving, promotes the development and application of BEV models, and improves the safety and reliability of autonomous driving technology.

[0109] Based on the bird's-eye view BEV data annotation method provided in the foregoing embodiments, this application also provides a bird's-eye view BEV data annotation device. Figure 6 This is a schematic diagram of a bird's-eye view BEV data annotation device provided in an embodiment of this application. Figure 6 As shown, the annotation device for the BEV data in this bird's-eye view includes:

[0110] The image information acquisition unit 601 is used to acquire the three-dimensional point cloud information of the parking environment collected by the vehicle-mounted lidar of the data acquisition vehicle, and the environmental surround view images of the parking environment collected by the vehicle-mounted surround view cameras at different locations, wherein the three-dimensional point cloud information and multiple different environmental surround view images are at the same timestamp.

[0111] The point cloud information acquisition unit 602 is used to obtain non-ground point clouds in the three-dimensional point cloud information based on the three-dimensional point cloud information; and to obtain obstacle information corresponding to each of the multiple different environmental surround view images based on the multiple different environmental surround view images.

[0112] The category point cloud acquisition unit 603 is used to perform matching processing on the non-ground point cloud based on the obstacle information corresponding to the multiple different environmental surround view images to obtain a categoryd non-ground point cloud.

[0113] The target point cloud acquisition unit 604 is used to filter the non-ground point cloud based on the vehicle-mounted lidar to obtain the target point cloud point that is closest to the vehicle-mounted lidar in the non-ground point cloud, wherein the target point cloud point indicates a ground boundary point.

[0114] The annotation result generation unit 605 is used to generate annotation results for BEV data based on the non-ground point cloud, the non-ground point cloud with categories, and the target point cloud points.

[0115] In one feasible implementation, the device further includes:

[0116] The transformation rule acquisition unit is used to acquire multiple coordinate transformation rules, one of which corresponds to the transformation rule between the vehicle-mounted LiDAR and a vehicle-mounted surround view camera.

[0117] The obstacle information includes obstacles corresponding to a first category and obstacles corresponding to a second category, wherein the first category focuses on the obstacle as a whole, and the second category focuses on the obstacle boundary; the category point cloud acquisition unit 603 includes:

[0118] The first category point cloud acquisition unit is used to, for each environmental panoramic image, based on the coordinate transformation rules corresponding to the environmental panoramic image, match the obstacles in the environmental panoramic image corresponding to the first category with the point cloud in the non-ground point cloud corresponding to the first category to obtain the non-ground point cloud corresponding to the first category.

[0119] The second category point cloud acquisition unit is used to, for each environmental panoramic image, based on the coordinate transformation rule corresponding to the environmental panoramic image, match the obstacles in the environmental panoramic image corresponding to the second category with the non-ground point cloud to obtain the non-ground point cloud corresponding to the second category.

[0120] The category-based point cloud acquisition unit is used to obtain a category-based non-ground point cloud based on the non-ground point cloud corresponding to the first category and the non-ground point cloud corresponding to the second category.

[0121] In one feasible implementation, the second type of point cloud acquisition unit is specifically used for:

[0122] The Z-value of the non-ground point cloud is adjusted to obtain a target Z-value of the non-ground point cloud, wherein the target Z-value is greater than the Z-value.

[0123] The obstacles in the environmental panoramic image corresponding to the second category are matched with the non-ground point cloud containing the target Z value to obtain the non-ground point cloud corresponding to the second category.

[0124] In one feasible implementation, the target point cloud point acquisition unit 604 is specifically used for:

[0125] Using the vehicle-mounted lidar as the center, a 360-degree angle range is determined;

[0126] For each angle within the 360-degree angle range, the non-ground point cloud at that angle is filtered to obtain the target point cloud point closest to the vehicle-mounted LiDAR in the non-ground point cloud at that angle.

[0127] In one feasible implementation, the annotation result generation unit 605 is specifically used for:

[0128] The non-ground point cloud is rasterized to obtain the first BEV;

[0129] The non-ground point cloud with categories is rasterized to obtain the second BEV;

[0130] The target point cloud points are rasterized to obtain the third BEV;

[0131] The second BEV and the third BEV are used as the labeling results of the first BEV.

[0132] In one feasible implementation, the point cloud information acquisition unit 602 is specifically used for:

[0133] The 3D point cloud information is classified based on a point cloud classification algorithm to obtain ground point cloud and non-ground point cloud in the 3D point cloud information;

[0134] Delete the ground point cloud from the three-dimensional point cloud information, and retain the non-ground point cloud from the three-dimensional point cloud information.

[0135] In one feasible implementation, the point cloud information acquisition unit 602 is specifically used for:

[0136] For each environmental surround view image, the environmental surround view image is segmented using an image segmentation model to obtain obstacle information in the environmental surround view image.

[0137] The bird's-eye view BEV data annotation device provided in this application embodiment has the same beneficial effects as the bird's-eye view BEV data annotation method provided in the above embodiment, which can improve annotation efficiency and annotation accuracy, and will not be described again here.

[0138] This application also provides a computer device. The computer device includes:

[0139] A memory that stores computer programs.

[0140] A processor is used to execute a computer program in memory to implement some or all of the steps in the bird's-eye view BEV data annotation method described in the foregoing embodiments.

[0141] This application also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the program implements some or all of the steps in the bird's-eye view BEV data annotation method described in the foregoing embodiments.

[0142] It should be noted that the "first" and "second" in the names such as "first" and "second" (if they exist) mentioned in the embodiments of this application are only used as name identifiers and do not represent the first and second in order.

[0143] It should also be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and equipment embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and equipment embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components indicated as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0144] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for labeling BEV data in a bird's-eye view, characterized in that, include: The data acquisition vehicle obtains 3D point cloud information of the parking environment collected by its onboard LiDAR and environmental surround view images of the parking environment collected by onboard surround view cameras at different locations, wherein the 3D point cloud information and multiple different environmental surround view images are at the same timestamp. Based on the three-dimensional point cloud information, obtain non-ground point cloud information in the three-dimensional point cloud information; and based on the multiple different environmental surround view images, obtain obstacle information corresponding to each of the multiple different environmental surround view images. Based on the obstacle information corresponding to the multiple different environmental surround view images, the non-ground point cloud is matched to obtain a non-ground point cloud with categories. Based on the vehicle-mounted lidar, the non-ground point cloud is filtered to obtain the target point cloud point that is closest to the vehicle-mounted lidar in the non-ground point cloud, wherein the target point cloud point indicates a ground boundary point. Based on the non-ground point cloud, the non-ground point cloud with categories, and the target point cloud points, the annotation results of BEV data are generated.

2. The method according to claim 1, characterized in that, Before performing matching processing on the non-ground point cloud based on the obstacle information corresponding to the multiple different environmental surround view images to obtain a categorized non-ground point cloud, the method further includes: Multiple coordinate transformation rules are obtained, one of which corresponds to the transformation rule between the vehicle-mounted LiDAR and a vehicle-mounted surround-view camera; The obstacle information includes obstacles corresponding to a first category and obstacles corresponding to a second category, wherein the first category focuses on the obstacle as a whole, and the second category focuses on the obstacle boundary; the matching process of the non-ground point cloud based on the obstacle information corresponding to the multiple different environmental surround view images to obtain a non-ground point cloud with categories includes: For each environmental panoramic image, based on the coordinate transformation rules corresponding to the environmental panoramic image, the obstacles in the environmental panoramic image corresponding to the first category are matched with the point cloud in the non-ground point cloud corresponding to the first category to obtain the non-ground point cloud corresponding to the first category. For each environmental panoramic image, based on the coordinate transformation rule corresponding to the environmental panoramic image, the obstacles in the environmental panoramic image corresponding to the second category are matched with the non-ground point cloud to obtain the non-ground point cloud corresponding to the second category; Based on the non-ground point cloud corresponding to the first category and the non-ground point cloud corresponding to the second category, a non-ground point cloud with categories is obtained.

3. The method according to claim 2, characterized in that, The step of matching the obstacles in the environmental surround view image corresponding to the second category with the non-ground point cloud to obtain the non-ground point cloud corresponding to the second category includes: The Z-value of the non-ground point cloud is adjusted to obtain a target Z-value of the non-ground point cloud, wherein the target Z-value is greater than the Z-value. The obstacles in the environmental panoramic image corresponding to the second category are matched with the non-ground point cloud containing the target Z value to obtain the non-ground point cloud corresponding to the second category.

4. The method according to claim 1, characterized in that, The step of filtering the non-ground point cloud based on the vehicle-mounted LiDAR to obtain the target point cloud point closest to the vehicle-mounted LiDAR in the non-ground point cloud includes: Using the vehicle-mounted lidar as the center, a 360-degree angle range is determined; For each angle within the 360-degree angle range, the non-ground point cloud at that angle is filtered to obtain the target point cloud point closest to the vehicle-mounted LiDAR in the non-ground point cloud at that angle.

5. The method according to claim 1, characterized in that, The annotation results for generating BEV data based on the non-ground point cloud, the categorized non-ground point cloud, and the target point cloud points include: The non-ground point cloud is rasterized to obtain the first BEV; The non-ground point cloud with categories is rasterized to obtain the second BEV; The target point cloud points are rasterized to obtain the third BEV; The second BEV and the third BEV are used as the labeling results of the first BEV.

6. The method according to claim 1, characterized in that, The step of obtaining non-ground point clouds from the three-dimensional point cloud information based on the three-dimensional point cloud information includes: The 3D point cloud information is classified based on a point cloud classification algorithm to obtain ground point cloud and non-ground point cloud in the 3D point cloud information; Delete the ground point cloud from the three-dimensional point cloud information, and retain the non-ground point cloud from the three-dimensional point cloud information.

7. The method according to claim 1, characterized in that, The step of obtaining obstacle information corresponding to each of the multiple different environmental surround view images includes: For each environmental surround view image, the environmental surround view image is segmented using an image segmentation model to obtain obstacle information in the environmental surround view image.

8. A device for labeling BEV data in a bird's-eye view, characterized in that, include: The image information acquisition unit is used to acquire the three-dimensional point cloud information of the parking environment collected by the vehicle-mounted lidar of the data acquisition vehicle, and the environmental surround view images of the parking environment collected by the vehicle-mounted surround view cameras at different locations, wherein the three-dimensional point cloud information and multiple different environmental surround view images are at the same timestamp. The point cloud information acquisition unit is used to obtain non-ground point clouds in the three-dimensional point cloud information based on the three-dimensional point cloud information; and to obtain obstacle information corresponding to each of the multiple different environmental surround view images based on the multiple different environmental surround view images. The category point cloud acquisition unit is used to match the non-ground point cloud based on the obstacle information corresponding to the multiple different environmental surround view images to obtain a category-based non-ground point cloud; The target point cloud acquisition unit is used to filter the non-ground point cloud based on the vehicle-mounted lidar to obtain the target point cloud point that is closest to the vehicle-mounted lidar in the non-ground point cloud, wherein the target point cloud point indicates a ground boundary point. The annotation result generation unit is used to generate annotation results for BEV data based on the non-ground point cloud, the non-ground point cloud with categories, and the target point cloud points.

9. A computer device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method for annotating bird's-eye view BEV data according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method for annotating bird's-eye view BEV data as described in any one of claims 1-7.

Images