Determination method and apparatus for obstacle information, and training method for sensing model

Abstract

Disclosed in the embodiments of the present disclosure are a determination method and apparatus for obstacle information, and a training method for a sensing model. The determination method for obstacle information comprises: acquiring point cloud data corresponding to each of the at least one time frame, and vehicle positioning information; on the basis of the point cloud data corresponding to each time frame, determining a target detection result and a semantic segmentation result that correspond to each time frame; on the basis of the point cloud data corresponding to each time frame, the vehicle positioning information, the target detection result and the semantic segmentation result, determining dense point cloud data corresponding to each time frame; and on the basis of the dense point cloud data corresponding to each time frame, determining obstacle information corresponding to each time frame. The embodiments of the present disclosure can implement automatic labeling of obstacle information, thereby improving the labeling working efficiency, reducing the manual labeling cost, and helping to provide rich sample data for model training.

Description

Obstacle information determination method, device and perception model training method

[0001] This disclosure claims priority to Chinese patent application number 202410110108X, filed on January 25, 2024, entitled “Method, device for determining obstacle information and training method for perception model”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method and device for determining obstacle information and a method for training a perception model. Background Art

[0003] With the continuous development of perception algorithms in scenarios such as autonomous driving and assisted driving, obstacle perception models based on elevation perception are gradually being applied. The perception results of obstacle perception models can provide richer information for vehicle planning and control decisions. For example, if the vehicle's drivable area is approximately flat, the bird's-eye view elevation map can directly reflect the feasibility of the area, the presence of obstacles, and the height of obstacles. However, in related technologies, training obstacle perception models requires manual annotation of obstacle information, such as the true elevation value, which is inefficient and costly. Summary of the Invention

[0004] In order to solve the above-mentioned technical problems such as low efficiency of manual labeling, the embodiments of the present disclosure provide a method and device for determining obstacle information and a method for training a perception model to improve the labeling efficiency of obstacle information.

[0005] A first aspect of the present disclosure provides a method for determining obstacle information, comprising: obtaining point cloud data and vehicle positioning information corresponding to each of at least one time frame; determining target detection results and semantic segmentation results corresponding to each of the time frames based on the point cloud data corresponding to each of the time frames; determining dense point cloud data corresponding to each of the time frames based on the point cloud data, the vehicle positioning information, the target detection results, and the semantic segmentation results corresponding to each of the time frames; and determining obstacle information corresponding to each of the time frames based on the dense point cloud data corresponding to each of the time frames.

[0006] A second aspect of the present disclosure provides a method for training an obstacle perception model, comprising: obtaining sensor data and obstacle information corresponding to each of at least one time frame, the sensor data including at least one of image data and point cloud data; obtaining the obstacle information based on the obstacle information determination method described in the first aspect; and training a target obstacle perception model based on the sensor data and obstacle information corresponding to each of the time frames to obtain a trained target obstacle perception model.

[0007] The third aspect of the present disclosure provides a device for determining obstacle information, including: a first acquisition module, used to acquire point cloud data and vehicle positioning information corresponding to each time frame in at least one time frame; a first processing module, used to determine the target detection results and semantic segmentation results corresponding to each time frame based on the point cloud data corresponding to each time frame; a second processing module, used to determine the dense point cloud data corresponding to each time frame based on the point cloud data, the vehicle positioning information, the target detection results and the semantic segmentation results corresponding to each time frame; and a third processing module, used to determine the obstacle information corresponding to each time frame based on the dense point cloud data corresponding to each time frame.

[0008] A fourth aspect of the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program for executing the method described in any of the above embodiments of the present disclosure.

[0009] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and the processor for reading the executable instructions from the memory and executing the instructions to implement the method described in any one of the above embodiments of the present disclosure.

[0010] A sixth aspect of the present disclosure provides a computer program product, which, when instructions in the computer program product are executed by a processor, executes the method provided in any of the above embodiments of the present disclosure.

[0011] Based on the obstacle information determination method, apparatus, and perception model training method provided by the above-mentioned embodiments of the present disclosure, by acquiring point cloud data and vehicle positioning information corresponding to each time frame in at least one time frame, target detection results and semantic segmentation results corresponding to each time frame can be determined based on the point cloud data corresponding to each time frame. Based on the point cloud data, vehicle positioning information, target detection results, and semantic segmentation results corresponding to each time frame, dense point cloud data corresponding to each time frame can be determined, and then obstacle information corresponding to each time frame can be determined based on the dense point cloud data corresponding to each time frame. The embodiments of the present disclosure can determine dense point cloud data for each time frame, and the dense point cloud data of each time frame can effectively contain obstacle information for that time frame. Therefore, the obstacle information for that time frame can be automatically determined based on the dense point cloud data, thereby achieving automated annotation of obstacle information, improving annotation efficiency, and reducing manual annotation costs. The obstacle information for each time frame obtained through automatic annotation can be used together with the sensor data for each time frame to train the obstacle perception model, helping to provide rich sample data for model training and effectively improving model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG1 is an exemplary application scenario of the method for determining obstacle information provided by the present disclosure;

[0013] FIG2 is a flow chart of a method for determining obstacle information provided by an exemplary embodiment of the present disclosure;

[0014] FIG3 is a flow chart of a method for determining obstacle information provided by another exemplary embodiment of the present disclosure;

[0015] FIG4 is a schematic diagram of a process for determining dense point cloud data provided by an exemplary embodiment of the present disclosure;

[0016] FIG5 is a flowchart of a method for determining obstacle information provided by yet another exemplary embodiment of the present disclosure;

[0017] FIG6 is a flowchart of a method for determining obstacle information provided by another exemplary embodiment of the present disclosure;

[0018] FIG7 is a flowchart of a method for determining obstacle information provided by yet another exemplary embodiment of the present disclosure;

[0019] FIG8 is a flowchart of determining a true elevation value according to an exemplary embodiment of the present disclosure;

[0020] FIG9 is a schematic diagram of a voxelization result provided by an exemplary embodiment of the present disclosure;

[0021] FIG10 is a schematic diagram of an elevation map from a bird's-eye view provided by an exemplary embodiment of the present disclosure;

[0022] FIG11 is a flow chart of a method for training an obstacle perception model according to an exemplary embodiment of the present disclosure;

[0023] FIG12 is a schematic structural diagram of an apparatus for determining obstacle information provided by an exemplary embodiment of the present disclosure;

[0024] FIG13 is a schematic structural diagram of an apparatus for determining obstacle information provided by another exemplary embodiment of the present disclosure;

[0025] FIG14 is a structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0027] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0028] Overview of the Disclosure

[0029] In the process of realizing the present disclosure, the inventors discovered that in scenarios such as autonomous driving and assisted driving, with the continuous development of perception algorithms, obstacle perception models based on elevation perception have gradually been applied. The perception results of the obstacle perception model can provide richer information for vehicle planning and control decisions. For example, the vehicle's drivable area is approximately flat, and the elevation map from a bird's-eye view can directly reflect the feasibility of the area, the existence of obstacles, and the height of obstacles. However, in related technologies, the training of obstacle perception models requires manual annotation of obstacle information, such as manual annotation of true elevation values, which is inefficient and costly.

[0030] Exemplary Overview

[0031] Figure 1 illustrates an exemplary application scenario of the obstacle information determination method provided by the present disclosure. As shown in Figure 1, when training an obstacle perception model, the obstacle information determination method of the present disclosure (which can be executed in the obstacle information determination device of the present disclosure) can be used to automatically label the sensor data corresponding to each time frame in at least one time frame, obtaining obstacle information corresponding to the sensor data in each time frame. The sensor data in each time frame is used as training sample data for the obstacle perception model, and the obtained obstacle information corresponding to the sensor data in each time frame is used as the label data (or true value) for each time frame. The obstacle perception model is then trained using the obstacle perception model training device to obtain a trained target obstacle perception model. The sensor data corresponding to each time frame can include at least one of image data and point cloud data. Specifically, obtaining the obstacle information can include obtaining point cloud data and vehicle positioning information corresponding to each time frame in the at least one time frame. Based on the point cloud data corresponding to each time frame, the target detection result and semantic segmentation result corresponding to each time frame can be determined. Based on the point cloud data, vehicle positioning information, target detection results, and semantic segmentation results corresponding to each time frame, the dense point cloud data corresponding to each time frame can be determined. Furthermore, based on the dense point cloud data corresponding to each time frame, the obstacle information corresponding to each time frame can be determined. Since the dense point cloud data of each time frame is obtained, the dense point cloud data of each time frame can effectively contain the obstacle information of that time frame. Therefore, the obstacle information of that time frame can be automatically determined based on the dense point cloud data, thereby achieving automated annotation of the obstacle information, improving annotation efficiency, and reducing manual annotation costs. Furthermore, the obstacle information of each time frame obtained through automatic annotation can be used together with the sensor data of each time frame to train the obstacle perception model, helping to provide rich sample data for model training to effectively improve model performance.

[0032] Exemplary Methods

[0033] FIG2 is a flow chart of a method for determining obstacle information provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as servers, terminal devices, and vehicle-mounted computing platforms. As shown in FIG2 , the method of the embodiment of the present disclosure may include the following steps:

[0034] Step 201: Obtain point cloud data and vehicle positioning information corresponding to each time frame in at least one time frame.

[0035] The number of time frames can be any number. In order to improve the accuracy of obstacle information, at least one time frame can generally include multiple time frames. Point cloud data can include point cloud data obtained based on any radar, such as lidar, millimeter wave radar, ultrasonic radar, etc. Point cloud data can also include point cloud data obtained based on other methods. Vehicle positioning information can include vehicle position information and posture information, and can also include other relevant information, such as speed, angular velocity, etc. Vehicle positioning information can be obtained through any positioning method. For example, the vehicle can be positioned based on a positioning device such as a wheel speed meter and a GPS (Global Positioning System) to obtain vehicle positioning information for each time frame.

[0036] In some optional embodiments, sensors on the vehicle can collect point cloud data of the vehicle's surroundings while the vehicle is driving, obtaining point cloud data for multiple consecutive time frames during the driving process, and obtaining vehicle positioning information corresponding to each time frame. The collected point cloud data can also be sampled at intervals to obtain point cloud data for multiple time frames and vehicle positioning information.

[0037] In some optional embodiments, the point cloud data may be point cloud data in a radar coordinate system. For example, the dimension of the point cloud data may be expressed as T×N×3, where T represents the number of time frames, N represents the number of points included in the point cloud data of each time frame, and 3 represents the three-dimensional coordinates x, y, and z of the radar coordinate system. The data dimension of the vehicle positioning information may be expressed as T×7, where 7 includes three-dimensional position coordinates and a posture represented by a quaternion (in actual applications, the posture may also be represented in other ways, not limited to quaternions). That is, the vehicle positioning information may include the vehicle posture information of each time frame in the T time frames, and the vehicle posture information may include the position information and posture information of the vehicle when the point cloud data of the time frame is collected. The vehicle posture information may be posture information in a global coordinate system, and the global coordinate system may be, for example, a world coordinate system or a coordinate system with the vehicle's starting position as the origin. The global coordinate system may be converted to the radar coordinate system of each time frame.

[0038] Step 202 : Based on the point cloud data corresponding to each time frame, determine the target detection result and semantic segmentation result corresponding to each time frame.

[0039] For each time frame, the target detection result corresponding to that time frame may include a 3D bounding box of the target object included in the point cloud data corresponding to that time frame. The number of target objects may be one or more. The semantic segmentation result corresponding to that time frame may include the semantic type corresponding to each point in the point cloud data of that time frame. Semantic types may include, for example, ground, tree, building, vehicle, pedestrian, etc., and may be set based on actual needs.

[0040] In some optional embodiments, a pre-acquired object detection model can be used to perform object detection on the point cloud data corresponding to each time frame to obtain object detection results corresponding to each time frame. The object detection model can be any applicable model, for example, the object detection model can be implemented based on various versions of pointpillars, and the present disclosure is not limited thereto.

[0041] In some optional embodiments, a pre-acquired semantic segmentation model can be used to perform semantic segmentation on the point cloud data corresponding to each time frame to obtain semantic segmentation results corresponding to each time frame. The semantic segmentation model can be any applicable model, for example, the semantic segmentation model can be implemented based on various versions of PointNet, and the present disclosure is not limited thereto.

[0042] In some optional embodiments, a multi-task model can also be used to process the point cloud data corresponding to each time frame, and simultaneously obtain the object detection results and semantic segmentation results corresponding to each time frame. The specific implementation method of the multi-task model is not limited.

[0043] Step 203 : Based on the point cloud data, vehicle positioning information, target detection results, and semantic segmentation results corresponding to each time frame, dense point cloud data corresponding to each time frame is determined.

[0044] For each time frame, the dense point cloud data corresponding to the time frame may be point cloud data obtained by densifying the point cloud data of the time frame.

[0045] In some optional embodiments, densification of the point cloud data of a time frame can be achieved by combining the point cloud data of that time frame and the frames before and after it. The point cloud data of adjacent time frames may overlap, so the point cloud data of the time frame can be supplemented with the point cloud data of the frames before and after it to achieve densification. Vehicle positioning information can be used to unify the coordinate systems of the point cloud data of different time frames to achieve point cloud data alignment.

[0046] In some optional embodiments, the dense point cloud data corresponding to each time frame may carry the semantic type of each point to facilitate the subsequent determination of obstacle information.

[0047] Step 204 : Determine obstacle information corresponding to each time frame based on the dense point cloud data corresponding to each time frame.

[0048] Among them, the dense point cloud data corresponding to each time frame can better contain the point cloud information of the object perceived in the time frame, so that the obstacle information corresponding to each time frame can be determined based on the dense point cloud data corresponding to each time frame.

[0049] In some optional embodiments, the obstacle information may include the elevation value of the obstacle (or the true elevation value), and may also include the type of obstacle and other relevant information, such as the size of the obstacle, which can be set according to actual needs.

[0050] The obstacle information determination method provided by this embodiment obtains point cloud data and vehicle positioning information corresponding to each time frame in at least one time frame. Based on the point cloud data corresponding to each time frame, the target detection results and semantic segmentation results corresponding to each time frame can be determined. Based on the point cloud data, vehicle positioning information, target detection results, and semantic segmentation results corresponding to each time frame, dense point cloud data corresponding to each time frame can be determined. Furthermore, based on the dense point cloud data corresponding to each time frame, obstacle information corresponding to each time frame can be determined. The disclosed embodiment can determine dense point cloud data for each time frame. The dense point cloud data for each time frame can effectively contain obstacle information for that time frame. Therefore, the obstacle information for that time frame can be automatically determined based on the dense point cloud data, thereby enabling automated annotation of obstacle information, improving annotation efficiency and reducing manual annotation costs. The obstacle information for each time frame obtained through automatic annotation can then be used together with the sensor data for each time frame to train an obstacle perception model, helping to provide rich sample data for model training and effectively improving model performance.

[0051] FIG3 is a flowchart of a method for determining obstacle information provided by another exemplary embodiment of the present disclosure.

[0052] In some optional embodiments, determining the dense point cloud data corresponding to each time frame based on the point cloud data, vehicle positioning information, target detection results, and semantic segmentation results corresponding to each time frame in step 203 may include:

[0053] Step 2031: For any time frame in each time frame, based on the point cloud data, target detection results, semantic segmentation results, and vehicle positioning information corresponding to the time frame and at least one other time frame before and after the time frame, determine the dense point cloud data corresponding to the time frame.

[0054] For each time frame, the selection of other time frames before and after the time frame can be determined based on the position of the time frame in the time frame sequence formed by each time frame. For example, for the starting time frame in the time frame sequence, one or more other time frames after the starting time frame can be determined to be used for point cloud densification of the starting time frame. For some time frames close to the starting time frame, such as the second time frame adjacent to the starting time frame, one or more other time frames after the starting time frame and the second time frame can be used for point cloud densification of the second time frame. For any time frame near the middle of the time frame sequence, if the number of other time frames before and after the time frame is large enough, such as the number of other time frames before and after the time frame are both greater than a number threshold, the same number of other time frames before and after the time frame (e.g., the same number as the number threshold) can be used for point cloud densification of the time frame. For example, the five other time frames before and after the time frame can be used for point cloud densification of the time frame.

[0055] In some optional embodiments, for each time frame, after determining the other time frames before and after the time frame, the point cloud data between the time frames can be aligned according to the vehicle positioning information corresponding to the time frame and the other time frames, so as to unify the point cloud data of the time frame and the other time frames into the same coordinate system, for example, they can be unified into the radar coordinate system of the time frame, or unified into the global coordinate system, so as to facilitate the addition of the point cloud data of the other time frames to the time frame, realize the densification of the point cloud data of the time frame, and obtain the dense point cloud data corresponding to the time frame.

[0056] For each time frame, this embodiment combines the point cloud data of other time frames before and after the time frame and the target detection results, semantic segmentation results and vehicle positioning information to determine the dense point cloud data of the time frame, thereby achieving effective densification of the point cloud data of each time frame, thereby better reflecting the obstacle information. Furthermore, the obstacle information of each time frame can be effectively obtained through the dense point cloud data, which helps to improve the accuracy of the obstacle information.

[0057] FIG4 is a schematic diagram of a process for determining dense point cloud data provided by an exemplary embodiment of the present disclosure.

[0058] In some optional embodiments, determining the dense point cloud data corresponding to the time frame in step 2031 based on the point cloud data, target detection results, semantic segmentation results, and vehicle positioning information corresponding to the time frame and at least one other time frame before and after the time frame may include:

[0059] Step 20311: take the time frame as the target time frame, and determine the moving object point set in the point cloud data of the target time frame based on the target detection result and semantic segmentation result corresponding to the target time frame.

[0060] Among them, moving objects may include other moving objects such as other vehicles, pedestrians, and animals in the surrounding area. Based on the target detection results and semantic segmentation results corresponding to the target time frame, it is possible to determine which points in the point cloud data of the target time frame belong to moving objects, thereby obtaining a moving object point set in the point cloud data of the target time frame.

[0061] Step 20312: Based on the moving object point set corresponding to the target time frame, the point cloud data of the target time frame is filtered for moving objects to obtain first point cloud data corresponding to the target time frame after filtering out the moving objects.

[0062] In some optional embodiments, since the position of a moving object in two adjacent time frames may be different due to movement, it is often impossible to accurately unify the point clouds between time frames into the same coordinate system when aligning the point clouds between time frames, which can easily have an adverse effect on the fusion of point clouds of multiple time frames. Therefore, based on the moving object point set corresponding to the target time frame, the point cloud data of the target time frame can be filtered for moving objects to obtain the first point cloud data corresponding to the target time frame after filtering out the moving objects. Similarly, based on the moving object point sets corresponding to each other time frame, the point cloud data of each other time frame can be filtered for moving objects to obtain the first point cloud data corresponding to each other time frame after filtering out the moving objects.

[0063] Step 20313: Based on the vehicle positioning information corresponding to the target time frame and each other time frame, the first point cloud data corresponding to the target time frame and each other time frame are fused to obtain fused point cloud data corresponding to the target time frame.

[0064] The fusion method of fusing the first point cloud data corresponding to the target time frame and each of the other time frames may include standardizing the coordinate system of the first point cloud data and merging the unified point cloud data.

[0065] In some optional embodiments, the first point cloud data of the target time frame and the first point cloud data of each other time frame can be converted to the same coordinate system based on the vehicle positioning information of the target time frame and the vehicle positioning information of each other time frame. For example, each first point cloud data can be converted to the vehicle coordinate system or radar coordinate system corresponding to the target time frame, or converted to a global coordinate system, so as to achieve the standardization of the coordinate systems of each first point cloud data. Specifically, based on the vehicle positioning information of any time frame, a conversion relationship between the vehicle coordinate system and the global coordinate system can be established. The radar coordinate system and the vehicle coordinate system have a fixed conversion relationship, thereby achieving mutual conversion between the radar coordinate system, the vehicle coordinate system and the global coordinate system. For example, the first point cloud data of other time frames can be converted from the radar coordinate system to the vehicle coordinate system of the other time frame, then from the vehicle coordinate system to the global coordinate system, and then from the global coordinate system to the radar coordinate system of the target time frame.

[0066] In some optional embodiments, the fused point cloud data of the target time frame can be point cloud data in the radar coordinate system corresponding to the target time frame, or point cloud data in the global coordinate system, and can be converted according to actual needs.

[0067] Step 20314: Determine the dense point cloud data corresponding to the target time frame based on the moving object point set of the target time frame and the fused point cloud data of the target time frame.

[0068] The moving object point set of the target time frame and the fused point cloud data of the target time frame can be unified into the same coordinate system and then fused to obtain dense point cloud data corresponding to the target time frame. For example, if the fused point cloud data of the target time frame and the moving object point set of the target time frame are both point cloud data in the radar coordinate system of the target time frame, the moving object point set of the target time frame can be added to the fused point cloud data to obtain dense point cloud data corresponding to the target time frame.

[0069] This embodiment first filters the moving object point set to obtain the first point cloud data after filtering the moving object in the target time frame, and then fuses the first point cloud data of the target time frame with the first point cloud data of each other time frame. This can reduce the adverse effects of the moving object on the point cloud fusion and avoid the moving object from generating a ghost in the target time frame. After fusion, the moving object point set of the target time frame is added to the fused point cloud data, which helps to improve the accuracy and effectiveness of the dense point cloud data.

[0070] In some optional embodiments, after obtaining the target detection results and semantic segmentation results corresponding to each time frame, the method of the embodiment of the present disclosure further includes: determining the moving object point set corresponding to each time frame based on the target detection results and semantic segmentation results corresponding to each time frame; filtering the point cloud data of each time frame for moving objects based on the moving object point set corresponding to each time frame to obtain the first point cloud data corresponding to each time frame after filtering out the moving objects. Then, in step 2031, determining the dense point cloud data corresponding to the time frame based on the point cloud data, target detection results, semantic segmentation results, and vehicle positioning information corresponding to the time frame and at least one other time frame before and after the time frame may include: taking the time frame as the target time frame, and based on the vehicle positioning information corresponding to the target time frame and each other time frame, fusing the first point cloud data corresponding to the target time frame and each other time frame to obtain fused point cloud data corresponding to the target time frame; and determining the dense point cloud data corresponding to the target time frame based on the moving object point set of the target time frame and the fused point cloud data of the target time frame.

[0071] In some optional embodiments, step 20313 of fusing the first point cloud data corresponding to the target time frame and each other time frame based on the vehicle positioning information corresponding to the target time frame and each other time frame to obtain fused point cloud data corresponding to the target time frame may include:

[0072] For any other time frame among the other time frames, based on the vehicle positioning information corresponding to the other time frame and the vehicle positioning information corresponding to the target time frame, determine the coordinate conversion relationship between the first point cloud data of the other time frame and the first point cloud data of the target time frame; based on the coordinate conversion relationship, convert the first point cloud data corresponding to the other time frame into the coordinate system corresponding to the first point cloud data of the target time frame to obtain the second point cloud data of the other time frame; fuse the second point cloud data corresponding to each other time frame with the first point cloud data of the target time frame to obtain the fused point cloud data corresponding to the target time frame.

[0073] The coordinate system corresponding to the first point cloud data can be the radar coordinate system or vehicle coordinate system of the target time frame, or it can be a global coordinate system. Taking the radar coordinate system as an example, for each other time frame, the vehicle position corresponding to the other time frame can be determined based on the vehicle positioning information corresponding to the other time frame. Then, based on the vehicle position, the conversion relationship between the vehicle coordinate system of the other time frame and the global coordinate system can be determined. Then, combined with the conversion relationship between the radar coordinate system and the vehicle coordinate system of the other time frame, the conversion relationship between the radar coordinate system of the other time frame and the global coordinate system can be determined. Similarly, the conversion relationship between the radar coordinate system of the target time frame and the global coordinate system can be obtained. Then, combined with the conversion relationship between the radar coordinate system of the target time frame and the global coordinate system and the conversion relationship between the radar coordinate system of the other time frame and the global coordinate system, the coordinate conversion relationship between the first point cloud data of the other time frame and the first point cloud data of the target time frame can be obtained. Based on this coordinate conversion relationship, the first point cloud data corresponding to the other time frame can be converted to the coordinate system corresponding to the first point cloud data of the target time frame to obtain the second point cloud data of the other time frame. After obtaining the second point cloud data for each of the other time frames, the second point cloud data for each of the other time frames can be fused with the first point cloud data for the target time frame to obtain fused point cloud data corresponding to the target time frame. The fusion of the second point cloud data with the first point cloud data for the target time frame can include data merging. Data merging involves merging points in the second point cloud data with points in the first point cloud data for the target time frame according to their specific coordinates into integrated point cloud data.

[0074] This embodiment realizes the unification of the coordinate system of the point cloud data by converting the first point cloud data of each other time frame into the coordinate system corresponding to the first point cloud data of the target time frame, thereby facilitating the effective fusion of the point cloud data of the target time frame with the other time frames and improving the accuracy and reliability of the fused point cloud data.

[0075] FIG5 is a flowchart of a method for determining obstacle information provided by yet another exemplary embodiment of the present disclosure.

[0076] In some optional embodiments, determining obstacle information corresponding to each time frame based on the dense point cloud data corresponding to each time frame in step 204 may include:

[0077] Step 2041 : Based on the dense point cloud data corresponding to each time frame, generate the true elevation value corresponding to each time frame.

[0078] For each time frame, the true elevation value corresponding to the time frame may include the elevation values ​​of the drivable area and obstacles in the time frame.

[0079] In some optional embodiments, the dense point cloud data of each time frame may contain relevant attribute information of objects surrounding the vehicle in the time frame. Therefore, the elevation value corresponding to the time frame can be generated based on the dense point cloud data corresponding to the time frame as the true elevation value corresponding to the time frame.

[0080] In some optional embodiments, the elevation truth values ​​corresponding to each time frame may be generated by voxelizing the dense point cloud data or by converting the dense point cloud data to a bird's-eye view.

[0081] Step 2042: Determine obstacle information corresponding to each time frame based on the true elevation value corresponding to each time frame.

[0082] In some optional embodiments, the true elevation values ​​corresponding to each time frame can be used as the obstacle information corresponding to each time frame. The true elevation value represents the elevation value of the obstacle. The specific obstacle can also be further determined based on the true elevation value as the obstacle information.

[0083] This embodiment generates the true elevation value corresponding to each time frame based on the dense point cloud data corresponding to each time frame, thereby achieving automatic labeling of the true elevation value, which helps to enrich the training samples of the elevation perception model.

[0084] FIG6 is a flowchart of a method for determining obstacle information provided by yet another exemplary embodiment of the present disclosure.

[0085] In some optional embodiments, step 2041 of generating the true elevation value corresponding to each time frame based on the dense point cloud data corresponding to each time frame may include:

[0086] In step 20411 , for any time frame among the time frames, the time frame is used as the target time frame, and the dense point cloud data corresponding to the target time frame is voxelized to obtain the occupancy result of the dense point cloud data in the voxel grid coordinate system.

[0087] The voxel grid coordinate system can be a three-dimensional grid coordinate system, with each voxel grid representing a three-dimensional grid. For example, a small cube of a certain volume can represent a certain volume of physical space, for example, each voxel grid represents 20 cubic centimeters of physical space. By voxelizing the dense point cloud data, the distribution of each point in the dense point cloud data in the voxel grid coordinate system can be determined. Based on the distribution of each point, the occupancy of the dense point cloud data in the voxel grid coordinate system can be determined. For example, the points distributed in each voxel grid can be determined based on the distribution. Some voxel grids may have no distributed points, while others may have one or more distributed points. A voxel grid with no distributed points can be considered unoccupied. For a voxel grid with multiple distributed points, a single point can be determined based on the multiple points and recorded corresponding to the voxel grid. For example, a point can be randomly determined from the multiple points and retained, while the other points can be removed. Alternatively, a mode point or a central point can be determined based on the multiple points and retained, without limitation. The specific situation where each voxel grid has no distribution points or one point is retained is taken as the occupancy result.

[0088] In some optional embodiments, the voxel grid coordinate system may be a uniform grid coordinate system or a non-uniform grid coordinate system. A uniform grid coordinate system, for example, a three-dimensional grid coordinate system in which each voxel grid is a small cube of equal volume. A non-uniform grid coordinate system, for example, a polar coordinate system, may be dense near the voxel grid and sparse far away. The specific voxel grid coordinate system is not limited.

[0089] In some optional embodiments, a three-dimensional bounding box of the dense point cloud data can be determined based on the dense point cloud data, and a voxel grid coordinate system can be determined based on the three-dimensional bounding box, that is, the space enclosed by the three-dimensional bounding box is divided into voxel grids. The voxel grid coordinate system can use any point in the space enclosed by the three-dimensional bounding box as its origin, for example, any vertex of the three-dimensional bounding box as its origin, without specific limitation. Alternatively, a voxel grid coordinate system of a preset size range can be set based on the empirical size of the dense point cloud data for each time frame.

[0090] Step 20412, based on the occupancy result, determine the true value of the elevation corresponding to the target time frame.

[0091] Among them, the occupancy result reflects the height information of each object in the target time frame in three-dimensional space. Therefore, the true value of the elevation corresponding to the target time frame can be determined based on the occupancy result.

[0092] This embodiment voxelizes the dense point cloud data of each time frame, and determines the elevation true value corresponding to each time frame based on the occupancy result of the dense point cloud data obtained by voxelization in the voxel grid coordinate system, thereby achieving effective determination of the elevation true value and improving the accuracy of the elevation true value.

[0093] In some optional embodiments, determining the true elevation value corresponding to the target time frame based on the occupancy result in step 20412 may include: converting the occupancy result from the voxel grid coordinate system to the target grid coordinate system to obtain the target occupancy result of the dense point cloud data in the target grid coordinate system; the physical height corresponding to each grid in the height direction of the target grid coordinate system includes the physical height corresponding to at least one voxel in the voxel grid coordinate system. For any grid in the target grid coordinate system, the number of points occupying the grid is determined based on the target occupancy result. Based on the number of points occupying the grid, the true value of the obstacle presence corresponding to the grid is determined. Based on the true value of the obstacle presence corresponding to each grid in the target grid coordinate system, the true elevation value corresponding to the target time frame is determined.

[0094] Among them, the target grid coordinate system can be a three-dimensional grid coordinate system. Each grid of the target grid coordinate system can correspond to a certain physical height in the height direction. For example, the height of each grid corresponds to a physical height of N centimeters (for example, 20 centimeters). The physical height corresponding to each grid of the target grid coordinate system can be greater than the voxel grid coordinate system, so as to divide the physical height represented by the voxel grid coordinate system into multiple height ranges. For example, the target grid coordinate system can be expressed as a grid coordinate system of L×W×H (for example, 512×512×20), where L represents the number of pixels in the length direction, W represents the number of pixels in the width direction, and H represents the number of height ranges. For example, the physical height of 4 meters is divided into 20 height ranges, and the height of each grid represents a physical height of 20 centimeters.

[0095] In some optional embodiments, the occupancy result can be converted from the voxel grid coordinate system to the target grid coordinate system based on the correspondence between the voxel grid coordinate system and the target grid coordinate system, thereby obtaining the target occupancy result in the target grid coordinate system. For example, a point retained by each voxel grid in which points are distributed in the voxel grid coordinate system is converted to the target grid coordinate system to obtain the target occupancy result. Since the physical space represented by each grid in the target grid coordinate system is larger than the voxel grid, one or more points may fall into a grid in the target grid coordinate system, i.e., the number of points occupying that grid is greater than 0. It is also possible that no points fall into some grids, i.e., the number of points occupying that grid is 0. For any grid in the target grid coordinate system, if the number of points occupying that grid is greater than 1, the true value of the obstacle presence corresponding to that grid can be determined to be 1. If the number of points occupying that grid is 0, the true value of the obstacle presence corresponding to that grid can be determined to be 0. Based on the true value of the obstacle presence corresponding to each grid in the target grid coordinate system, the true elevation value corresponding to the target time frame can be obtained. For example, the true value of the elevation corresponding to the pixel position can be obtained based on the true value of the obstacle presence in the height direction of each pixel position in the L×W pixels in the target grid coordinate system. For example, if the true value of the obstacle presence in the h grids below the pixel position is 1, and the true value of the obstacle presence in the grid above is 0, it can be said that an obstacle exists in the physical space of the h grids below. The physical height represented by the h grids below can then be determined as the true value of the elevation at the pixel position. Alternatively, the true value of the obstacle presence corresponding to each grid in the target grid coordinate system can be used as the true value of the elevation corresponding to the target time frame. That is, the true value of the obstacle presence in the L×W×H three-dimensional structure is determined as the obstacle information corresponding to the target time frame.

[0096] In some optional embodiments, the physical length corresponding to the length of each grid in the target grid coordinate system may be greater than or equal to the physical length corresponding to the length of each voxel grid in the voxel grid coordinate system. The physical width corresponding to the width of each grid in the target grid coordinate system may be greater than or equal to the physical width corresponding to the width of each voxel grid in the voxel grid coordinate system. For example, each voxel grid in the voxel grid coordinate system represents 20 cubic centimeters of physical space, and each grid in the target grid coordinate system represents 8,000 cubic centimeters of physical space of 20 cm × 20 cm × 20 cm.

[0097] In some optional embodiments, the target grid coordinate system may be a uniform grid coordinate system or a non-uniform grid coordinate system, and the uniformity of the target grid coordinate system may be consistent with or inconsistent with the voxel grid coordinate system.

[0098] This embodiment converts the occupancy results into the target grid coordinate system, which facilitates the discretization of the physical height represented by the voxel grid coordinate system, obtains the true value of the existence of obstacles in grids of different height ranges under the target grid coordinate system, and helps to quickly obtain the true value of the elevation corresponding to the target time frame.

[0099] FIG7 is a flowchart of a method for determining obstacle information provided by yet another exemplary embodiment of the present disclosure.

[0100] In some optional embodiments, step 2041 of generating the true elevation value corresponding to each time frame based on the dense point cloud data corresponding to each time frame may include:

[0101] Step 2041a: For any time frame among the time frames, the time frame is taken as the target time frame, and based on the dense point cloud data of the target time frame, the distribution result of each point in the dense point cloud data in the two-dimensional grid coordinate system of the bird's-eye view corresponding to the target time frame is determined.

[0102] The two-dimensional grid coordinate system can be a coordinate system that grids any plane perpendicular to the z-axis of the coordinate system in which the dense point cloud data resides. For example, if the dense point cloud data is a radar coordinate system, the plane at z = 0 can be gridded to form a two-dimensional grid coordinate system from a bird's-eye view. The distribution result can include the points corresponding to each two-dimensional grid in the two-dimensional grid coordinate system. For any two-dimensional grid, the number of points distributed in the two-dimensional grid can be zero, one, or multiple.

[0103] In some optional embodiments, each point in the dense point cloud data may be projected into a two-dimensional grid coordinate system to obtain a distribution result of the two-dimensional grid coordinate system corresponding to a bird's-eye view of each point in the target time frame.

[0104] Step 2041b: Based on the distribution results, determine the true value of the elevation from the bird's-eye view corresponding to the target time frame.

[0105] In some optional embodiments, based on the distribution results, each two-dimensional grid distribution point of the two-dimensional grid coordinate system can be determined, and the elevation true value corresponding to the two-dimensional grid can be determined according to each two-dimensional grid distribution point, so that the elevation true value corresponding to each two-dimensional grid can be used as the elevation true value of the bird's-eye view corresponding to the target time frame.

[0106] In some optional embodiments, for each time frame, the true value of the bird's-eye view elevation corresponding to that time frame can be represented as a single-channel elevation map of L'×W'×1. For example, a 512×512×1 elevation map, where 512×512 can represent the length and width pixel range, and the elevation value of the single channel can represent the height value corresponding to each length and width pixel.

[0107] This embodiment distributes dense point cloud data to a two-dimensional grid coordinate system from a bird's-eye view, and then determines the true value of the elevation from a bird's-eye view corresponding to the target time frame based on the distribution of the dense point cloud data in the two-dimensional grid coordinate system. This achieves effective determination of the two-dimensional elevation map, thereby providing rich sample data for the training of the obstacle perception model from a bird's-eye view.

[0108] In some optional embodiments, determining the distribution results of each point in the dense point cloud data of the target time frame in the two-dimensional grid coordinate system of the bird's-eye view corresponding to the target time frame in step 2041a may include: projecting each point in the dense point cloud data of the target time frame to the plane where the two-dimensional grid coordinate system is located, and obtaining the distribution results of each point in the two-dimensional grid coordinate system.

[0109] Among them, according to the three-dimensional coordinates of each point in the dense point cloud data, the z coordinate can be set to the z coordinate of the plane where the two-dimensional grid coordinate system is located to obtain the projection point of each point on the plane. According to the conversion relationship between the point coordinates of the plane and the grid of the two-dimensional grid coordinate system, the two-dimensional grid to which each projection point belongs can be determined, and the distribution results of each point in the two-dimensional grid coordinate system can be obtained.

[0110] This embodiment projects each point of the dense point cloud data onto the plane of the two-dimensional grid coordinate system to obtain the distribution result of the point in the two-dimensional grid coordinate system, which facilitates the determination of the true value of the elevation from the bird's-eye view corresponding to the target time frame.

[0111] In some optional embodiments, determining the true elevation value of the bird's-eye view corresponding to the target time frame based on the distribution results in step 2041b may include: determining the highest point corresponding to each two-dimensional grid in the two-dimensional grid coordinate system based on the distribution results; using the elevation value of the highest point in each two-dimensional grid as the true elevation value corresponding to the two-dimensional grid; and determining the true elevation value of the bird's-eye view corresponding to the target time frame based on the true elevation values ​​corresponding to each two-dimensional grid.

[0112] The distribution result may include the points distributed in each two-dimensional grid in the two-dimensional grid coordinate system. Based on the three-dimensional coordinates of the points distributed in each two-dimensional grid, the highest point of the two-dimensional grid distribution, such as the point with the largest z coordinate, can be obtained. Thus, the elevation value of the highest point can be used as the true elevation value corresponding to the two-dimensional grid. The elevation value corresponding to each point can be the height value of the point, such as the z coordinate value, or the height value of the z coordinate value relative to the z coordinate value of the lowest point in the dense point cloud data. The specific setting can be based on actual needs.

[0113] In some optional embodiments, if the two-dimensional grid has no distribution points, the elevation value of the two-dimensional grid can be set to a preset value. The preset value can be, for example, a value representing a null value. The principle of setting the preset value is to have a large difference from the conventional elevation value to avoid confusion with the conventional elevation value. For example, the preset value can be set to 255.

[0114] This embodiment can determine the highest point corresponding to each two-dimensional grid through the distribution results, and use the elevation value of the highest point as the true elevation value corresponding to the two-dimensional grid, thereby effectively determining the true elevation value from a bird's-eye view and improving the accuracy of the true elevation value.

[0115] In some optional embodiments, FIG8 is a flowchart of a true elevation determination process provided by an exemplary embodiment of the present disclosure. As shown in FIG8 , the ego-vehicle trajectory may include vehicle positioning information for each time frame in at least one time frame. The radar point cloud data may include point cloud data corresponding to each time frame. After acquiring the radar point cloud data, semantic segmentation results and target detection results may be determined based on the radar unit data. Based on the semantic segmentation results and target detection results, moving object points in the radar point cloud data are removed. The radar point cloud data after removing the moving object points is then densified based on the ego-vehicle trajectory to obtain fused point cloud data corresponding to each time frame. The moving object points for each time frame are then added to the fused point cloud data to obtain dense point cloud data corresponding to each time frame. The dense point cloud data may then be voxelized into a three-dimensional structure to obtain occupancy results of the dense point cloud data in a voxel grid coordinate system. Based on the occupancy results, the true elevation values ​​corresponding to each time frame are determined. The dense point cloud data may also be converted into a bird's-eye view elevation map to obtain the true elevation values ​​corresponding to each time frame from a bird's-eye view. The specific operations of each step can be found in the above embodiments and will not be described in detail here.

[0116] In some optional examples, Figure 9 is a schematic diagram of voxelization results provided by an exemplary embodiment of the present disclosure. As shown in Figure 9, the left side is a schematic diagram of the voxel grid coordinate system for voxelization, and the right side is a visualization diagram of the elevation truth value after the occupancy results of the dense point cloud data in the voxel grid coordinate system are converted to the target grid coordinate system. Different grayscales can represent different elevation truth values.

[0117] In some optional examples, Figure 10 is a schematic diagram of an elevation map from a bird's-eye view provided by an exemplary embodiment of the present disclosure. As shown in Figure 10, different grayscales represent different true elevation values.

[0118] The above-mentioned embodiments of the present disclosure may be implemented individually or in any combination without conflict. The specific configuration may be based on actual needs and is not limited by the present disclosure.

[0119] Any of the obstacle information determination methods provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, any of the obstacle information determination methods provided in the embodiments of the present disclosure can be executed by a processor, such as by invoking corresponding instructions stored in a memory to execute any of the obstacle information determination methods mentioned in the embodiments of the present disclosure. This will not be further described below.

[0120] FIG11 is a flow chart of a method for training an obstacle perception model according to an exemplary embodiment of the present disclosure. This embodiment can be applied to electronic devices, such as servers, terminal devices, and the like. As shown in FIG11 , the method according to the embodiment of the present disclosure may include the following steps:

[0121] Step 301: Obtain sensor data and obstacle information corresponding to each time frame in at least one time frame.

[0122] The sensor data may include at least one of image data and point cloud data. The obstacle information may be obtained based on the obstacle information determination method provided in any of the above embodiments.

[0123] The point cloud data and obstacle information may refer to the above embodiments. The image data may be data collected synchronously with the point cloud data.

[0124] Step 302 : Based on the sensor data and obstacle information corresponding to each time frame, the obstacle perception model to be trained is trained to obtain a trained target obstacle perception model.

[0125] Among them, depending on the obstacle information, the obstacle perception model to be trained can be a model for predicting the obstacle information of the above-mentioned three-dimensional structure or a model for predicting the obstacle information from a bird's-eye view. For example, it can be an elevation perception model from a bird's-eye view, which can be set according to actual needs.

[0126] In some optional embodiments, the input to the obstacle perception model to be trained can be multi-sensor data such as surround view images from multiple perspectives, lidar point clouds, ultrasonic radar point clouds, etc. For multi-sensor data, feature extraction can be performed separately and then fused in the bird's-eye view space.

[0127] In some optional embodiments, during training, sensor data from a single time frame can be used as a training sample. Specifically, a single time frame of sensor data is input, and the obstacle information for that time frame serves as the label data for that time frame. Because obstacle information for each time frame training sample is derived from both the previous and next frames, the trained model can be enriched, effectively improving model performance.

[0128] The obstacle perception model training method provided in this embodiment uses the obstacle information obtained through automatic annotation in the above embodiment as label data to train the obstacle perception model to be trained, which can effectively increase the richness of training samples, thereby helping to improve the performance of the model.

[0129] Any obstacle perception model training method provided in the embodiments of the present disclosure can be executed by any appropriate device with data processing capabilities, including but not limited to a terminal device and a server. Alternatively, any obstacle perception model training method provided in the embodiments of the present disclosure can be executed by a processor, such as by invoking corresponding instructions stored in a memory to execute any obstacle perception model training method mentioned in the embodiments of the present disclosure. This will not be further described below.

[0130] Exemplary devices

[0131] Figure 12 is a schematic diagram of the structure of an obstacle information determination device provided by an exemplary embodiment of the present disclosure. The device of this embodiment can be used to implement the corresponding method embodiment of the present disclosure. The device shown in Figure 12 may include: a first acquisition module 501, a first processing module 502, a second processing module 503, and a third processing module 504.

[0132] The first acquisition module 501 may be configured to acquire point cloud data and vehicle positioning information corresponding to each time frame in at least one time frame.

[0133] The first processing module 502 may be configured to determine target detection results and semantic segmentation results corresponding to each time frame based on the point cloud data corresponding to each time frame;

[0134] The second processing module 503 may be configured to determine dense point cloud data corresponding to each time frame based on the point cloud data, vehicle positioning information, target detection results, and semantic segmentation results corresponding to each time frame;

[0135] The third processing module 504 may be configured to determine obstacle information corresponding to each time frame based on the dense point cloud data corresponding to each time frame.

[0136] FIG13 is a schematic structural diagram of an obstacle information determination device provided by another exemplary embodiment of the present disclosure.

[0137] In some optional embodiments, the second processing module 503 may include: a first processing unit 5031, which can be used to determine, for any time frame in each time frame, the dense point cloud data corresponding to the time frame based on the point cloud data, target detection results, semantic segmentation results, and vehicle positioning information corresponding to the time frame and at least one other time frame before and after the time frame.

[0138] In some optional embodiments, the first processing unit 5031 can be specifically used to: take the time frame as the target time frame, and determine the moving object point set in the point cloud data of the target time frame based on the target detection result and semantic segmentation result corresponding to the target time frame. Based on the moving object point set corresponding to the target time frame, the point cloud data of the target time frame is filtered for moving objects to obtain the first point cloud data corresponding to the target time frame after filtering out the moving objects. Based on the vehicle positioning information corresponding to the target time frame and each other time frame, the first point cloud data corresponding to the target time frame and each other time frame are fused to obtain the fused point cloud data corresponding to the target time frame. Based on the moving object point set of the target time frame and the fused point cloud data of the target time frame, the dense point cloud data corresponding to the target time frame is determined.

[0139] In some optional embodiments, the first processing unit 5031 can be specifically used to: for any other time frame in each other time frame, based on the vehicle positioning information corresponding to the other time frame and the vehicle positioning information corresponding to the target time frame, determine the coordinate conversion relationship between the first point cloud data of the other time frame and the first point cloud data of the target time frame; based on the coordinate conversion relationship, convert the first point cloud data corresponding to the other time frame to the coordinate system corresponding to the first point cloud data of the target time frame to obtain the second point cloud data of the other time frame; fuse the second point cloud data corresponding to each other time frame with the first point cloud data of the target time frame to obtain the fused point cloud data corresponding to the target time frame.

[0140] In some optional embodiments, the third processing module 504 may include: a second processing unit 5041 and a third processing unit 5042 .

[0141] The second processing unit 5041 may be configured to generate a true elevation value corresponding to each time frame based on the dense point cloud data corresponding to each time frame.

[0142] The third processing unit 5042 may be configured to determine obstacle information corresponding to each time frame based on the true elevation value corresponding to each time frame.

[0143] In some optional embodiments, the second processing unit 5041 may be specifically configured to: for any time frame among the time frames, use the time frame as a target time frame, voxelize the dense point cloud data corresponding to the target time frame, obtain an occupancy result of the dense point cloud data in a voxel grid coordinate system, and determine the true elevation value corresponding to the target time frame based on the occupancy result.

[0144] In some optional embodiments, the second processing unit 5041 can be specifically used to: convert the occupancy result from the voxel grid coordinate system to the target grid coordinate system to obtain the target occupancy result of the dense point cloud data in the target grid coordinate system; the physical height corresponding to each grid in the height direction of the target grid coordinate system includes the physical height corresponding to at least one voxel in the voxel grid coordinate system. For any grid in the target grid coordinate system, based on the target occupancy result, the number of points occupying the grid is determined. Based on the number of points occupying the grid, the true value of the existence of an obstacle corresponding to the grid is determined. Based on the true value of the existence of an obstacle corresponding to each grid in the target grid coordinate system, the true value of the elevation corresponding to the target time frame is determined.

[0145] In some optional embodiments, the second processing unit 5041 may be specifically configured to: for any time frame among the time frames, use the time frame as a target time frame, and determine, based on the dense point cloud data of the target time frame, a distribution result of each point in the dense point cloud data in a two-dimensional grid coordinate system at a bird's-eye view corresponding to the target time frame. Based on the distribution result, determine a true value of the elevation at the bird's-eye view corresponding to the target time frame.

[0146] In some optional embodiments, the second processing unit 5041 can be specifically used to: project each point in the dense point cloud data of the target time frame to the plane where the two-dimensional grid coordinate system is located, and obtain the distribution result of each point in the two-dimensional grid coordinate system.

[0147] In some optional embodiments, the second processing unit 5041 may be specifically configured to: determine, based on the distribution result, the highest point corresponding to each two-dimensional grid in the two-dimensional grid coordinate system; use the elevation value of the highest point in each two-dimensional grid as the true elevation value corresponding to the two-dimensional grid; and determine the true elevation value of the bird's-eye view corresponding to the target time frame based on the true elevation values ​​corresponding to each two-dimensional grid.

[0148] The above-mentioned embodiments of the present disclosure may be implemented individually or in any combination without conflict. The specific configuration may be based on actual needs and is not limited by the present disclosure.

[0149] The beneficial technical effects corresponding to the exemplary embodiment of this device can be found in the corresponding beneficial technical effects of the above exemplary method part, which will not be repeated here.

[0150] Exemplary electronic devices

[0151] FIG14 is a structural diagram of an electronic device provided by an embodiment of the present disclosure, which includes at least one processor 11 and a memory 12 .

[0152] The processor 11 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0153] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the one or more computer program instructions to implement the methods and / or other desired functions of the various embodiments of the present disclosure described above.

[0154] In one example, the electronic device 10 may further include an input device 13 and an output device 14 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0155] The input device 13 may also include, for example, a keyboard, a mouse, etc.

[0156] The output device 14 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0157] Of course, for simplicity, FIG14 only shows some of the components of the electronic device 10 related to the present disclosure, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may further include any other appropriate components depending on the specific application.

[0158] Exemplary computer program products and computer-readable storage media

[0159] In addition to the above methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to perform the steps in the methods of various embodiments of the present disclosure described in the above "Exemplary Method" section.

[0160] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0161] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps in the method of various embodiments of the present disclosure described in the above “Exemplary Method” section.

[0162] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or component comprising electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0163] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be considered as essential to each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0164] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include these modifications and variations.

Claims

1. A method for determining obstacle information, comprising: Obtaining point cloud data and vehicle positioning information respectively corresponding to each of at least one time frame; Based on the point cloud data respectively corresponding to each of the time frames, determining target detection results and semantic segmentation results respectively corresponding to each of the time frames; Based on the point cloud data, the vehicle positioning information, the target detection results, and the semantic segmentation results respectively corresponding to each of the time frames, determining dense point cloud data respectively corresponding to each of the time frames; Based on the dense point cloud data respectively corresponding to each of the time frames, determining obstacle information respectively corresponding to each of the time frames.

2. The method according to claim 1, wherein, The step of determining the dense point cloud data respectively corresponding to each of the time frames based on the point cloud data, the vehicle positioning information, the target detection results, and the semantic segmentation results respectively corresponding to each of the time frames includes: For any one of the time frames, based on the point cloud data, the target detection results, the semantic segmentation results, and the vehicle positioning information respectively corresponding to this time frame and at least one other time frame before and after this time frame, determining the dense point cloud data corresponding to this time frame.

3. The method according to claim 2, wherein The step of determining the dense point cloud data corresponding to this time frame based on the point cloud data, the target detection results, the semantic segmentation results, and the vehicle positioning information respectively corresponding to this time frame and at least one other time frame before and after this time frame includes: Taking this time frame as a target time frame, and based on the target detection results and the semantic segmentation results corresponding to the target time frame, determining a set of moving object points in the point cloud data of the target time frame; Based on the set of moving object points corresponding to the target time frame, filtering the moving objects from the point cloud data of the target time frame to obtain first point cloud data after filtering the moving objects corresponding to the target time frame; Based on the vehicle positioning information respectively corresponding to the target time frame and each of the other time frames, fusing the first point cloud data respectively corresponding to the target time frame and each of the other time frames to obtain fused point cloud data corresponding to the target time frame; Based on the set of moving object points of the target time frame and the fused point cloud data of the target time frame, determining the dense point cloud data corresponding to the target time frame.

4. The method according to claim 3, wherein, The step of fusing the first point cloud data respectively corresponding to the target time frame and each of the other time frames based on the vehicle positioning information respectively corresponding to the target time frame and each of the other time frames to obtain fused point cloud data corresponding to the target time frame includes: For any one of the other time frames, based on the vehicle positioning information corresponding to this other time frame and the vehicle positioning information corresponding to the target time frame, determining a coordinate transformation relationship between the first point cloud data of this other time frame and the first point cloud data of the target time frame; Based on the coordinate conversion relationship, convert the first point cloud data corresponding to the other time frame to the coordinate system corresponding to the first point cloud data of the target time frame, and obtain the second point cloud data of the other time frame; Fuse the second point cloud data corresponding to each of the other time frames with the first point cloud data of the target time frame to obtain the fused point cloud data corresponding to the target time frame.

5. According to the method described in any one of claims 1-4, wherein Determining the obstacle information corresponding to each of the time frames based on the dense point cloud data corresponding to each of the time frames includes: Generating the elevation ground truth corresponding to each of the time frames based on the dense point cloud data corresponding to each of the time frames; Determining the obstacle information corresponding to each of the time frames based on the elevation ground truth corresponding to each of the time frames.

6. The method according to claim 5, wherein, Generating the elevation ground truth corresponding to each of the time frames based on the dense point cloud data corresponding to each of the time frames includes: For any one of the time frames in each of the time frames, use this time frame as the target time frame, voxelize the dense point cloud data corresponding to the target time frame, and obtain the occupancy result of the dense point cloud data in the voxel grid coordinate system; Determine the elevation ground truth corresponding to the target time frame based on the occupancy result.

7. The method according to claim 6, wherein, Determining the elevation ground truth corresponding to the target time frame based on the occupancy result includes: Convert the occupancy result from the voxel grid coordinate system to the target grid coordinate system to obtain the target occupancy result of the dense point cloud data in the target grid coordinate system; the physical height corresponding to each grid in the height direction of the target grid coordinate system includes the physical height corresponding to at least one voxel of the voxel grid coordinate system; For any grid in the target grid coordinate system, determine the number of points occupying the grid based on the target occupancy result; Determine the truth value of the existence of an obstacle corresponding to the grid based on the number of points occupying the grid; Determine the elevation ground truth corresponding to the target time frame based on the truth values of the existence of obstacles corresponding to each of the grids in the target grid coordinate system.

8. The method according to claim 5, wherein Generating the elevation ground truth corresponding to each of the time frames based on the dense point cloud data corresponding to each of the time frames includes: For any one of the time frames in each of the time frames, use this time frame as the target time frame, and based on the dense point cloud data of the target time frame, determine the distribution result of each point in the dense point cloud data in the two-dimensional grid coordinate system corresponding to the bird's-eye view of the target time frame; Determine the elevation ground truth corresponding to the bird's-eye view of the target time frame based on the distribution result.

9. The method according to claim 8, wherein Determining the elevation ground truth corresponding to the bird's-eye view of the target time frame based on the distribution result includes: Based on the distribution result, determine the highest point corresponding to each two-dimensional grid in the two-dimensional grid coordinate system; Take the elevation value of the highest point in each two-dimensional grid as the elevation ground truth corresponding to the two-dimensional grid; Determine the elevation ground truth corresponding to the bird's-eye view of the target time frame based on the elevation ground truths corresponding to each of the two-dimensional grids.

10. The method according to claim 8, wherein, Determining the distribution result of each point in the dense point cloud data in the two-dimensional grid coordinate system corresponding to the bird's-eye view of the target time frame based on the dense point cloud data of the target time frame includes: Projecting each point in the dense point cloud data of the target time frame onto the plane where the two-dimensional grid coordinate system is located to obtain the distribution result of each point in the two-dimensional grid coordinate system.

11. A method for training an obstacle perception model, including: Obtaining sensor data and obstacle information respectively corresponding to each of at least one time frame, where the sensor data includes at least one of image data and point cloud data; The obstacle information is obtained based on the method for determining obstacle information according to any one of claims 1-10; Based on the sensor data and the obstacle information respectively corresponding to each time frame, training the obstacle perception model to be trained to obtain a trained target obstacle perception model.

12. A device for determining obstacle information, including: A first acquisition module, configured to acquire point cloud data and vehicle positioning information respectively corresponding to each of at least one time frame; A first processing module, configured to determine a target detection result and a semantic segmentation result respectively corresponding to each time frame based on the point cloud data respectively corresponding to each time frame; A second processing module, configured to determine dense point cloud data respectively corresponding to each time frame based on the point cloud data, the vehicle positioning information, the target detection result, and the semantic segmentation result respectively corresponding to each time frame; A third processing module, configured to determine obstacle information respectively corresponding to each time frame based on the dense point cloud data respectively corresponding to each time frame.

13. A computer-readable storage medium, where the storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1-10 or the method according to claim 11 above.

14. An electronic device, where the electronic device includes: A processor; A memory for storing executable instructions of the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1-10 or the method according to claim 11 above.

Images