Method, device, medium and vehicle for determining three-dimensional shape information of road obstacles

Abstract

The present invention discloses a method, device, medium and vehicle for determining three-dimensional shape information of road obstacles, which are applied to the field of autonomous driving technology. The method includes obtaining the features of the point cloud data and image data of the road area to be detected that are rasterized under the target perspective. The features after the fusion of the point cloud features and the image features are subjected to target detection processing to obtain the initial three-dimensional shape information of the obstacles in the road area to be detected. The current spatial voxel weight is continuously updated according to the new elevation information of the grid corresponding to each frame of point cloud data during the vehicle's forward movement, and the elevation information and geometric shape data of the grid in the obstacle area are correspondingly updated based on the updated spatial voxel weight, so as to realize the continuous updating of the initial three-dimensional shape information. The present invention can solve the problem that the related technology cannot accurately calculate the position of road obstacles, and can accurately determine the three-dimensional shape information of the pothole area and the low raised obstacle area of ​​the road.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method for determining three-dimensional shape information of a road obstacle, an electronic device, a non-volatile storage medium, a computer program product, and an unmanned vehicle. Background Art

[0002] With the rapid development of artificial intelligence and Internet of Things technologies, autonomous driving technology has also developed accordingly. The smoothness of the road not only affects the normal operation of autonomous vehicles and the comfort of passengers, but also affects the life of the road and its infrastructure.

[0003] When monitoring whether there are potholes and low raised obstacles on the road, the relevant technology cannot accurately identify the pothole areas and raised obstacle areas, and thus cannot accurately determine the locations of potholes and low raised obstacles on the road.

[0004] In view of this, improving the accuracy of determining three-dimensional topographic information of road potholes and low raised obstacle areas on the road is a technical problem that needs to be solved by technical personnel in this field.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present invention provides a method for determining three-dimensional shape information of road obstacles, an electronic device, a non-volatile storage medium, a computer program product and an unmanned vehicle, which can accurately determine the three-dimensional shape information of road pothole areas and road low raised obstacle areas.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In one aspect, the present invention provides a method for determining three-dimensional shape information of a road obstacle, comprising:

[0009] Acquire the features of the point cloud data and image data of the road surface area to be detected in the target perspective after rasterization; wherein the road surface area to be detected in the rasterization includes a plurality of grids, and each grid is provided with a spatial voxel in a horizontal direction perpendicular to the road surface;

[0010] The point cloud features and image features are fused, and the initial three-dimensional shape information of the obstacles in the road area to be detected is obtained by performing target detection processing on the fused features;

[0011] According to the new elevation information of the grid corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement, the spatial voxel weight at the current moment is updated, and the elevation information and geometric morphology data of the grid in the obstacle area are updated based on the updated spatial voxel weight, so as to continuously update the initial three-dimensional shape information.

[0012] In a first exemplary embodiment, the step of acquiring the features of the rasterized point cloud data and image data of the road surface area to be detected under the target viewing angle includes:

[0013] When an obstacle is detected in the area ahead of the vehicle, the position of the obstacle is monitored during the forward movement of the vehicle;

[0014] When the obstacle enters the road surface area to be detected, the road surface area to be detected in front of the current vehicle is rasterized to generate a bird's-eye view grid area including a plurality of grids, each grid having spatial voxels in a horizontal direction perpendicular to the road surface;

[0015] The point cloud data and image data of the bird's-eye view grid area are acquired, and the point cloud data and image data are converted to a bird's-eye view to obtain point cloud features and image features of the corresponding view.

[0016] The maximum distance between the to-be-detected road surface area and the vehicle along the vehicle's driving direction is less than or equal to the minimum distance between the area in front of the vehicle and the vehicle.

[0017] In a second exemplary embodiment, the initial three-dimensional shape information of obstacles in the road area to be detected is obtained by performing target detection processing on the fused features, including:

[0018] Pre-constructing and training a three-dimensional target detection network model; the three-dimensional target detection network model includes a detection task head, a classification task head and an instance segmentation task head, the detection task head is used to identify obstacles and output obstacle position information; the classification task head is used to identify obstacle category information; the instance segmentation task head is used to segment obstacles on the target plane to obtain instance segmentation results;

[0019] Inputting the fused features into the three-dimensional target detection network model to obtain the location information, category information and instance segmentation results of the obstacles;

[0020] Based on the instance segmentation result and the point cloud data in the obstacle area, determining the geometric description data of the obstacle through the three-dimensional surface fitting result;

[0021] The position information, category information and geometric description data of the obstacle are used as the initial three-dimensional shape information of the obstacle.

[0022] In a third exemplary embodiment, after updating the elevation information and geometric data of the grid in the obstacle area based on the updated spatial voxel weight, the method further includes:

[0023] Based on the transformation relationship between the vehicle coordinate system and the local world coordinate system, the three-dimensional shape information of the obstacle in the road area to be detected is converted into the local world coordinate system;

[0024] The local world coordinate system is the coordinate system of the road surface on which the vehicle is traveling.

[0025] In a fourth exemplary embodiment, before the transformation relationship between the vehicle coordinate system and the local world coordinate system is described, the method further includes:

[0026] Get the point cloud data and local map point cloud at the current moment;

[0027] Determine the transformation relationship between the vehicle coordinate system and the local world coordinate system by performing point cloud matching on the point cloud data and the local map point cloud;

[0028] The local map point cloud is data generated by splicing point cloud data at different times along the time axis.

[0029] In a fifth exemplary implementation, the spatial voxel weight at the current moment is updated according to the new grid elevation information corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement, including:

[0030] Obtaining the spatial voxels of each grid of the rasterized road surface area to be detected and the corresponding initial weight values;

[0031] The elevation information of the grid corresponding to the point cloud data of the current frame is obtained as new elevation information, and the weight value of the spatial voxel at the corresponding position is updated using the new elevation information.

[0032] In a sixth exemplary implementation, the updating of the spatial voxel weight at the current moment according to the new grid elevation information corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement includes:

[0033] Obtain the type of each grid area in the rasterized road surface area to be detected;

[0034] When the current grid belongs to the first type of obstacle area identified in the previous frame, if the current grid does not belong to the radar point cloud projection area, the weight of the spatial voxel of the current grid is determined based on the spatial voxel coordinate position, elevation information of each grid in the first type of obstacle area and the coordinate position of the current grid; if the current grid belongs to the radar point projection area, the weight of the spatial voxel of the current grid is determined based on the spatial voxel coordinate position, elevation information of each grid in the first type of obstacle area, the elevation information of the radar point projection area and the coordinate position of the current grid;

[0035] When the current grid belongs to the second type of obstacle area identified by the current frame, if the current grid does not belong to the radar point cloud projection area, the weight of the spatial voxel of the current grid is determined based on the spatial voxel coordinate position of each grid in the second type of obstacle area; if the current grid belongs to the radar point projection area, the weight of the spatial voxel of the current grid is determined based on the spatial voxel coordinate position of each grid in the second type of obstacle area and the elevation information of the radar point projection area.

[0036] In a seventh exemplary implementation, before obtaining the types of each grid area in the gridded road surface area to be detected, the method further includes:

[0037] Based on the one-dimensional Gaussian function relationship, a weight calculation relationship is generated according to the preset Gaussian curve control parameter value, the center position coordinates determined by different grid area types, and the grid space voxel position information as variables, so as to determine the weight value of each grid space voxel by calling the weight calculation relationship.

[0038] In an eighth exemplary implementation, updating elevation information of a grid in an obstacle area based on the updated spatial voxel weight includes:

[0039] Pre-training an elevation classification network, the elevation classification network is used to convert the depth estimation problem into a classification problem, and the elevation classification network determines a loss function based on obstacle type, elevation truth encoding information, obstacle area mask information, point cloud features and image features of the obstacle area grid, and spatial voxel weights of the obstacle area grid;

[0040] The elevation information of the grid in the obstacle area is obtained through the elevation classification network.

[0041] In a ninth exemplary implementation, the updating of the current spatial voxel weight according to the new grid elevation information corresponding to the current frame point cloud data during the vehicle's forward movement includes:

[0042] Determine the grids included in the obstacle area of ​​the current frame according to the position information of the obstacle identified in the previous frame in the bird's-eye view grid area at the current moment and the position information of the obstacle identified in the current frame in the bird's-eye view grid area at the current moment;

[0043] The elevation information of the grid included in the obstacle area of ​​the current frame is obtained, and the spatial voxel weights of the grid included in the obstacle area of ​​the current frame are updated.

[0044] In a tenth exemplary implementation, determining the grid included in the obstacle area of ​​the current frame according to the position information of the obstacle identified in the previous frame in the bird's-eye view grid area at the current moment and the position information of the obstacle identified in the current frame in the bird's-eye view grid area at the current moment includes:

[0045] According to the obstacle recognition results corresponding to the previous frame and the current frame, the grid in the corresponding obstacle area is determined;

[0046] Obtaining the position information of the obstacle detected in the area ahead of the vehicle at the current moment in the bird's-eye view grid area; the minimum distance between the area ahead of the vehicle and the vehicle along the vehicle's driving direction is greater than or equal to the maximum distance between the road surface area to be detected and the vehicle;

[0047] According to the position change of the vehicle in the local world coordinate system in the adjacent frame time period, the position information of the grid in the obstacle area of ​​the previous frame is updated to serve as the position information of the grid area of ​​the bird's-eye view of the obstacle identified in the previous frame at the current moment;

[0048] If the current point position is the position information of the obstacle identified in the previous frame at the current moment in the bird's-eye view grid area, or the position information of the obstacle identified in the current frame at the current moment in the bird's-eye view grid area, or the position information of the obstacle identified in the front driving area at the current moment in the bird's-eye view grid area, then a first preset value is set for the current point as a mask value; if the current point position is not the position information of the obstacle identified in the previous frame at the current moment in the bird's-eye view grid area, or the position information of the obstacle identified in the current frame at the current moment in the bird's-eye view grid area, or the position information of the obstacle identified in the front driving area at the current moment in the bird's-eye view grid area, then a second preset value is set for the current point as a mask value;

[0049] Obstacle area mask information is generated according to the mask value of each point in the road area to be detected, so as to identify the grids included in the obstacle area of ​​the current frame.

[0050] The present invention also provides an electronic device, comprising a processor, wherein the processor is used to implement the steps of the method for determining three-dimensional shape information of road obstacles as described in any of the preceding items when executing a computer program stored in a memory.

[0051] The present invention also provides a non-volatile storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for determining three-dimensional shape information of road obstacles as described in any of the preceding items are implemented.

[0052] The present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method for determining three-dimensional shape information of road obstacles as described in any of the preceding items.

[0053] Finally, the present invention also provides an unmanned vehicle, including an automatic driving perception system, wherein the automatic driving perception system is used to implement the steps of the method for determining three-dimensional morphological information of road obstacles as described in any of the preceding items when executing a computer program stored in a memory.

[0054] The advantage of the technical solution provided by the present invention is that obstacles are identified through target detection methods, and initial three-dimensional shape data of obstacles are obtained. Multiple frames of sensor data are collected as the vehicle moves forward, and each frame of data can obtain new elevation information for each grid. As the obstacle approaches the vehicle, the accuracy of obtaining sensor data will become higher and higher. The weights of the grid space voxels are continuously updated according to the new elevation information, and more accurate elevation information and obstacle geometry data can be gradually obtained through the update of weights, thereby accurately determining the three-dimensional shape information of potholes and low raised obstacle areas on the road, which is not only conducive to ensuring the safe and stable operation of autonomous driving vehicles and effectively improving the riding experience of passengers, but also conducive to rapid road repairs and extending the life of roads and their infrastructure.

[0055] In addition, the present invention also provides corresponding electronic devices, non-volatile storage media, computer program products and unmanned vehicles for the method of determining the three-dimensional morphology information of road obstacles, further making the method more practical, and the electronic devices, non-volatile storage media, computer program products and unmanned vehicles have corresponding advantages.

[0056] The technical features mentioned above, the technical features to be mentioned below, and the technical features shown separately in the drawings can be arbitrarily combined with each other, as long as the combined technical features are not contradictory. All feasible feature combinations are technical contents clearly recorded in this article. Any of the multiple sub-features contained in the same sentence can be applied independently, and does not have to be applied together with other sub-features. It should be understood that the above general description and the detailed description below are only exemplary and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A schematic diagram of a process for determining three-dimensional shape information of a road obstacle provided by the present invention;

[0059] Figure 2 A schematic diagram of a road surface area to be detected after rasterization provided by the present invention;

[0060] Figure 3 A schematic diagram of coordinate system definition in an exemplary example is provided for the present invention;

[0061] Figure 4 A schematic diagram of spatial voxels on a longitudinal section of a road surface provided by the present invention;

[0062] Figure 5 A side view schematic diagram of elevation estimation provided by the present invention;

[0063] Figure 6 A schematic diagram of a mask of an obstacle provided by the present invention;

[0064] Figure 7 A schematic diagram of the spatial voxel weights of obstacles provided by the present invention;

[0065] Figure 8 A schematic diagram of geometric description data of an obstacle provided by the present invention;

[0066] Fig. 9 A schematic flow chart of another method for determining three-dimensional shape information of road obstacles provided by the present invention;

[0067] Fig.10 A structural diagram of a specific implementation of the device for determining three-dimensional shape information of road obstacles provided by the present invention;

[0068] Fig.11 A structural diagram of a specific implementation of the electronic device provided by the present invention;

[0069] Fig.12 A schematic diagram of obstacle recognition for an unmanned vehicle provided by the present invention. DETAILED DESCRIPTION

[0070] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, the terms "first", "second", "third", "fourth", etc. in the specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations of the two are intended to cover non-exclusive inclusions. The term "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described here as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

[0071] It is understandable that potholes or low raised obstacles on the road can cause the autonomous vehicle to drive unsteadily and then bump, which will not only affect the comfort of the passengers, but also cause the vehicle to suddenly lose control and cause traffic accidents. If the vehicle is driven on potholes and low raised obstacles for a long time, it will damage the vehicle's suspension system, tires, and sensors, which will affect the vehicle's performance and safety. In addition, potholes on the road will damage the road material and cause the settlement of the roadbed. If they are not repaired in time, the potholes will easily expand and deepen, accelerating the damage to the road infrastructure and affecting the service life and carrying capacity of the road. In order to avoid the above problems, during the driving process of the vehicle, it will monitor in real time whether there are potholes or raised obstacle areas on the road ahead, remind the driver of the road detection situation, and issue a warning to slow down the speed of the car, or absorb and offset the impact and vibration caused by potholes / low raised obstacles on the road through the intelligent suspension system.

[0072] At present, the traditional way to monitor whether there are potholes or low raised obstacles on the road is manual visual inspection, where structural engineers and certified inspectors regularly check whether there are potholes or low raised obstacles on the road surface and report the location of such target areas. This method relies on the personal experience of inspectors and engineers, which not only cannot guarantee the accuracy of the detection results, but is also inefficient, costly, and dangerous. In order to solve the drawbacks of manual visual inspection, related technologies use computer vision technology to achieve more efficient and safer identification and detection of road obstacles such as potholes or low obstacles.

[0073] Related technologies use computer vision algorithms to achieve three-dimensional road imaging through 3D point cloud modeling, and then detect road obstacle areas based on segmentation methods to improve the accuracy of road target area detection. This method uses some algorithms, such as RANSAC (Random Sample Consensus) to improve the robustness of road point cloud modeling and DCNN (Deep Convolution Neural Network) to improve image classification accuracy. However, this type of method requires a small field of view and is based on a single-frame 3D road point cloud that is a plane or quadratic surface. The detection accuracy and scene adaptability are still difficult to achieve practicality, and it is difficult to obtain the geometric characteristics of road pits such as width, depth, and volume with high precision, and thus it is impossible to accurately identify and calculate the three-dimensional morphological information of road obstacles. For example, a related technology realizes the detection of potholes on the road surface of unmanned vehicles through multi-sensor fusion. This method uses sound waves to scan the road surface ahead. When a pothole is detected, the camera is used to capture the road image, and then the pre-processed image is threshold segmented to obtain a binary image. The number of white pixels in the binary image is counted to determine whether there is water in the pothole. The image information captured by the camera is used to calculate the width of the pothole edge from the road edge on both sides, and the minimum torque for the unmanned vehicle to pass through the pothole is calculated. Finally, the relationship between the minimum torque and the maximum torque is used to determine whether the unmanned vehicle can pass safely. This method is greatly affected by the threshold value for image binarization, making it difficult to adapt to various lighting conditions, and it is unable to provide information such as the depth of the pothole, so it is impossible to accurately calculate the three-dimensional shape information of the pothole area and low obstacles. Another related technology uses an RGB-D (RedGreenBlue-Depth, a color image with depth information) depth camera to detect low raised obstacles and potholes on the road. This method uses the original point cloud of the RGB-D depth camera for preprocessing to obtain the corresponding plane point cloud, extracts the raised low obstacle point cloud data from the plane point cloud, and projects the extracted raised low obstacle point cloud onto the plane model parameter plane, downsamples the raised low obstacle point cloud on the model parameter plane, performs convex hull segmentation, transforms the convex hull to the / map (map) coordinate system and publishes it, merges the projected raised low obstacle point cloud into the ground plane point cloud to form a new plane point cloud, transforms the new plane point cloud to the base_footprint (robot body coordinate system) into a ground plane two-dimensional grid map, performs pseudo pothole filtering on the ground plane two-dimensional grid map, and extracts potholes using the edge extraction method.Since the effective working distance of RGB-D cameras is very limited, usually within 10 meters, and they are basically ineffective under strong outdoor light, the accuracy of the point cloud generated by the RGB-D camera is poor due to the influence of the projection density of the generator. Not only is the use scenario limited, but the effectiveness of the actual use process needs to be verified, and the calculation accuracy of the three-dimensional morphological information of potholes and low obstacles cannot be ensured. Another related technology is based on neural networks and binocular stereo vision to detect potholes on the road. This method uses a binocular camera installed in front of the vehicle to obtain the left and right views of the road scene; and uses the target detection neural network to detect the left view. When the existence of a pothole is detected, the left and right views are stereo matched to obtain a depth map, and the pothole position information predicted by the neural network is used to cut out the depth map of the pothole and the road surface around the pothole. Finally, a method combining RANSAC and genetic algorithms is used to fit the road plane around the pothole, so as to calculate the distance from the pothole point cloud to the fitting plane, and the distance is summed and averaged to obtain the pothole depth. The minimum circumscribed rectangle of the projection point is determined by projecting the pothole point cloud onto the fitting road plane, and the area of ​​the minimum circumscribed rectangle is determined to obtain the area of ​​the pothole. However, due to the existence of void points in parallax, this method has large estimation errors in the depth direction, making it difficult to obtain high-precision pothole depths and bump heights, and thus unable to accurately identify pothole areas and low obstacles, nor to accurately determine the three-dimensional morphology information of pothole areas and low obstacles.

[0074] In view of this, the present invention determines the three-dimensional shape information of the obstacle through point cloud data and image data, and then as the vehicle continues to move forward, the area where the obstacle is located continues to approach the vehicle, and the weights of the spatial voxels are continuously updated with the new elevation information, so that the accuracy of the elevation information and geometric data can be continuously improved, thereby effectively improving the accuracy of low obstacle / pothole recognition and three-dimensional shape information calculation. After introducing the technical scheme of the present invention, various non-limiting embodiments of the present invention are described in detail below. In order to better illustrate the present invention, numerous specific details are given in the specific embodiments below. Those skilled in the art should understand that the present invention can also be implemented without these specific details. In other examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present invention.

[0075] First see Figure 1 , Figure 1 A schematic flow chart of a method for determining three-dimensional shape information of a road obstacle provided in this embodiment, this embodiment may include the following contents:

[0076] S101: Acquire the features of the rasterized point cloud data and image data of the road area to be detected under the target viewing angle.

[0077] In this embodiment, the road surface area to be detected is located in front of the vehicle, and is a space area extending forward from the position of the front of the vehicle. Its size can be flexibly selected according to actual conditions. The vehicle can be any vehicle with an autonomous driving function, and is not limited to autonomous driving vehicles or unmanned vehicles. The vehicle can also be called an ego vehicle (the current autonomous driving vehicle). In the field of computer vision, a network structure is used to predict elevation information to replace monocular depth estimation, such as BEVheight and RoadBEV (road surface reconstruction in a bird's-eye view). However, the error in the predicted elevation of this method can reach about 2 cm, and this error makes it impossible to accurately judge low raised obstacles and potholes. The present invention divides the road surface area to be detected into multiple grids, that is, rasterization processing, such as Figure 2 As shown in the figure, the road area to be detected in front of the vehicle is divided into multiple grids under the target perspective. Since the laser radar installed on the vehicle will emit laser light during the vehicle's driving process, Figure 2 The black area in represents the projection point of the laser radar point cloud on the road surface, and the white area represents no data. The grid size and the number of grids can be flexibly selected according to the actual situation, such as the amount of calculation, the obstacle detection accuracy and the configuration parameters of the laser radar installed on the vehicle, which does not affect the implementation of the present invention. Among them, the obstacle is an object that the present invention needs to detect in the road surface area to be detected, and the object has a small height in the direction perpendicular to the horizontal plane of the road surface, including but not limited to low raised obstacles, such as speed bumps, raised cracks, and potholes, such as puddles, shallow pits, and deep pits. Taking into account that there may be a certain angle between the road surface and the xy plane of the vehicle coordinate system, if this situation is not taken into account, the error will be introduced into the elevation calculation. For areas with smaller elevations, that is, obstacles such as low raised obstacles and potholes, misjudgment may be caused. Therefore, the present invention also provides each grid with a spatial voxel in a horizontal direction perpendicular to the road surface. The spatial voxel is a plurality of spatial cubes arranged in a horizontal direction perpendicular to the road surface. The spatial voxel can reflect the ups and downs of the ground, which is used in the subsequent accurate calculation of the elevation information of low raised obstacles and potholes, so as to be used for roadside perception, smart transportation, V2X (vehicle to everything, information exchange between vehicles and the outside world), smart security, collaborative perception and other fields for obstacle avoidance, environmental perception, etc.

[0078] Among them, the point cloud data is the point cloud data collected by the vehicle laser radar, and the image data is the image collected by the vehicle camera during the vehicle driving process. The acquired point cloud data and image data can be converted to the target perspective. The target perspective can be determined according to the method used for the actual three-dimensional shape information. For example, the target perspective is a bird's-eye view. For the convenience of description, the features obtained after the conversion are defined as point cloud features and image features. In order to improve the accuracy of determining the three-dimensional shape information of road obstacles, before acquiring the point cloud data and image data, the calibration of each sensor of the vehicle can be completed through the internal mechanism of the vehicle. The sensor includes but is not limited to an inertial measurement unit, an odometer, a laser radar, and a forward camera. The coordinate system transformation relationship between the sensors is obtained. For example, the calibration parameters before the vehicle leaves the factory can be directly used. In order to further improve the accuracy of generating three-dimensional shape information of road obstacles, the time alignment of point cloud data and image data can be ensured through a data time synchronization mechanism.

[0079] S102: Fusing the point cloud features and the image features, and performing target detection processing on the fused features to obtain initial three-dimensional shape information of obstacles in the road area to be detected.

[0080] After the multimodal features under the target perspective are obtained in the previous step, in order to further improve the recognition accuracy of obstacles, such as low obstacles and potholes, point cloud features and image features can be fused. Based on the fused features, any target detection algorithm, such as R-CNN (Region-based Convolutional Neural Networks), Fast R-CNN (a fast region-based convolutional neural network), Faster R-CNN (a faster region-based convolutional neural network) and YOLOv3 (You Only Look Once version 3, network model name), is used to perform target detection on the fused features. The target refers to the obstacle to be identified in this embodiment. The obstacle in the fused features identified by the target detection algorithm can further determine the elevation information, position, and geometric data of the obstacle, such as direction, length, width, and area, and use it as the three-dimensional shape information of the identified obstacle. Since the three-dimensional shape information determined in this step is not accurate, the three-dimensional shape information will be continuously updated through the data collected by the vehicle continuously approaching the obstacle, so this step defines it as the initial three-dimensional shape information.

[0081] S103: According to the new elevation information of the grid corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement, the spatial voxel weight at the current moment is updated, and the elevation information and geometric data of the grid in the obstacle area are updated based on the updated spatial voxel weight to continuously update the initial three-dimensional shape information.

[0082] When step S101 divides the road surface area to be detected into multiple grids with spatial voxels, and each grid is provided with spatial voxels, the inclination angle between the road surface area to be detected and the horizontal plane of the vehicle coordinate system can be determined by the plane fitting method. For example, the point cloud data of each grid is plane fitted to obtain the corresponding three-dimensional plane equation, and the road surface normal vector is obtained according to the three-dimensional plane equation. The angle between the road surface normal vector and the horizontal plane of the vehicle coordinate system is the inclination angle. According to the installation position of the laser radar and the structural parameters of the vehicle, the distance from the origin of the laser radar coordinate system to the ground can be obtained. Based on the principle of triangular geometry, the x-coordinate of the intersection of the vehicle coordinate system X-axis and the road surface can be obtained according to the distance and the inclination angle. , the elevation information of the corresponding grid can be obtained according to the difference between the coordinate and the horizontal coordinate of the grid and the inclination angle. For the classic LSS (Lattice Sampling and Segmentation, a technology used for three-dimensional point cloud processing and scene understanding), in order to estimate the depth information of each pixel, it is generally believed that for each pixel, the corresponding grid is set along the light ray, and each grid does not have a weight setting similar to a priority. In order to accurately calculate the three-dimensional shape information of the obstacle, this embodiment can further set weights for the spatial voxels. As the vehicle continues to move forward, that is, it continues to approach low obstacles / potholes, the on-board sensor will continuously collect point cloud data. For each frame of point cloud data, new elevation information of each grid can be obtained, and the weight is continuously updated according to the new elevation information. Through the iterative mechanism, more accurate elevation information is gradually obtained, so that the elevation information and geometric data are continuously updated by updating the weights, and more accurate three-dimensional shape data is obtained. The target elevation information calculation method based on point cloud data in this embodiment can effectively improve the depth calculation accuracy compared to LSS. In addition, the depth or elevation estimation method based on LSS has high requirements on the accuracy of the labeled data and is difficult to label. However, this embodiment can further improve the depth prediction accuracy by using a regression method based on the initial true value with a certain accuracy, without the need for a large amount of labeled data.

[0083] In the technical solution provided in this embodiment, obstacles are identified through a target detection method, and initial three-dimensional shape data of the obstacles are obtained. Multiple frames of sensor data are collected as the vehicle moves forward, and each frame of data can obtain new elevation information for each grid. As the obstacle approaches the vehicle, the accuracy of obtaining sensor data will become higher and higher. The weights of the grid space voxels are continuously updated based on the new elevation information, and more accurate elevation information and obstacle geometry data can be gradually obtained through the update of the weights, thereby accurately determining the three-dimensional shape information of the potholes and low raised obstacle areas on the road, which is not only conducive to ensuring the safe and stable operation of the autonomous driving vehicle, effectively improving the riding experience of the passengers, but also conducive to rapid road maintenance and extending the life of the road and its infrastructure.

[0084] After the above embodiments determine the three-dimensional shape information of the obstacle, in order to play a corresponding role in roadside perception scenarios, smart traffic scenarios, V2X (vehicle to everything, i.e., vehicle-to-external information exchange) scenarios, smart security scenarios, and collaborative perception scenarios, such as accurately assessing the harmfulness of low raised obstacles / potholes, such as obstacle avoidance during automatic route planning. The three-dimensional shape information of the obstacle in the road area to be detected can also be converted to the local world coordinate system based on the transformation relationship between the vehicle coordinate system and the local world coordinate system.

[0085] In this embodiment, for data processing, the present invention also defines the coordinate system involved accordingly, such as Figure 3 As shown in the figure, the vehicle includes a far-field camera, a near-field camera, a lidar, an odometer, and an inertial measurement unit. For the convenience of drawing, the far-field camera and the near-field camera are not distinguished, and the odometer and inertial measurement unit are not drawn. Figure 3 middle, is the camera coordinate system, Along the camera optical axis. is the standard camera coordinate system, which is Along Rotation, where Vertically downward. is the laser radar coordinate system, is the local world coordinate system, which is defined on the road surface, where Vertically upward, For horizontal forward, For horizontal left, the vehicle coordinate system can usually adopt the laser radar coordinate system, and the local world coordinate system is the coordinate system of the road where the vehicle is driving. Exemplarily, the process of obtaining the transformation relationship between the vehicle coordinate system and the local world coordinate system can be: obtaining the point cloud data and the local map point cloud at the current moment; determining the transformation relationship between the vehicle coordinate system and the local world coordinate system by matching the point cloud data and the local map point cloud; wherein the local map point cloud is the data generated by splicing the point cloud data at different moments along the time axis. Of course, in order to further improve the accuracy of the three-dimensional morphology information, when determining the transformation relationship between coordinates, the point cloud data can be first densified, and when obtaining the laser radar point cloud data, the laser radar point cloud data can be distorted and corrected by the vehicle posture data of the odometer and the inertial measurement unit. The point cloud matching algorithm can adopt any algorithm, such as ICP (Iterative Closest Point, iterative closest point algorithm) and NDT (Normal Distribution Transform, normal distribution transformation), which does not affect the implementation of the present invention.

[0086] The above embodiment does not limit how to perform rasterization processing on the road surface area to be detected. Based on the above embodiment, the present invention also provides an exemplary rasterization processing implementation method of the road surface area to be detected, which may include the following contents:

[0087] As an exemplary rasterization method, the number of laser radar lines installed on the autonomous driving vehicle can be obtained; the number of grids and the grid size are determined according to the number of laser radar lines, and the spatial size of the road surface area to be detected is determined according to the grid data and the grid size; according to the number of grids and the grid size, the road surface area to be detected within the corresponding spatial size in front of the driving direction is rasterized.

[0088] As another exemplary rasterization method, the resolution of the laser radar installed on the autonomous driving vehicle can be obtained; the number of grids and the grid size are determined according to the resolution of the laser radar, and the spatial size of the road surface area to be detected is determined according to the grid data and the grid size; according to the number of grids and the grid size, the road surface area to be detected within the corresponding spatial size in front of the driving direction is rasterized.

[0089] In this embodiment, the number of grids in the bird's-eye view grid area and the size of the space corresponding to each grid on the road to be detected can be flexibly adjusted according to the number of laser radar lines or resolution. For example, the vehicle is equipped with a 32-line mechanical laser radar. Considering the accuracy and calculation amount, the number of grids in the bird's-eye view grid area is 2000×2000, and the space size corresponding to each grid is 0.01 meter×0.01 meter. The length and width of the space area corresponding to the bird's-eye view grid area are 20 meters and 20 meters respectively. For more than 20 meters, it is a far-field detection area. If the vehicle is equipped with a 128-line mechanical laser radar, the size of the space corresponding to the bird's-eye view grid area can be increased to 40 meters×30 meters. In order to improve the detection and recognition accuracy, the size of the space corresponding to each grid can be set to 0.01 meter×0.01 meter.

[0090] In order to further improve the recognition accuracy of road obstacles, based on the above embodiments, the present invention further provides a method for determining parameters of spatial voxels of a grid, which may include the following contents:

[0091] Obtain the maximum elevation of obstacles in the road area to be detected; divide the maximum elevation of obstacles according to a preset accuracy threshold to determine the total number of spatial cubes contained in the spatial voxels of each grid; determine the real physical space size corresponding to the spatial cube of the spatial voxels of each grid based on the maximum elevation of obstacles, the total number of spatial cubes and the preset plane size of spatial cubes.

[0092] In this embodiment, if Figure 4 As shown, Figure 4 The left side is a schematic diagram of the space voxel, which sets 2t space cubes, and the right side is the longitudinal section of the road surface. is the elevation difference between the road surface where the spatial voxel is located and the coordinate system of the autonomous driving vehicle, that is, the horizontal plane of the vehicle coordinate system. When the road surface is horizontal or the pitch angle between the vehicle coordinate system and the road surface is 0, 0. However, since the road surface itself may be inclined or when the vehicle passes through a low raised obstacle or a pothole area, there is a certain pitch angle between the vehicle coordinate system and the ground. In order to more accurately obtain the elevation of the low raised obstacle or pothole area, Values ​​such as Figure 4 As shown in the figure on the right, Take the elevation difference between the road surface where the spatial voxel is located and the horizontal plane of the vehicle coordinate system.

[0093] The number of space cubes contained in the space voxel is determined by the maximum elevation of the obstacle to be identified, such as a low raised obstacle / pothole area. For example, the total number of space cubes can be determined by the multiple of the number of segments of the maximum elevation of the obstacle, and the real physical space height corresponding to the space cube can be determined by the ratio of the maximum elevation of the obstacle to the total number of space cubes. The real physical space size corresponding to the space cube of the space voxel of each grid is determined according to the real physical space height and the grid size. For example, the space voxel may include A space cube, The maximum elevation of the low raised obstacle / pothole area to be identified is set. For example, the maximum elevation of the low raised obstacle / pothole area to be identified is 0.15 meters. In order to accurately identify the elevation information of each obstacle such as a low raised obstacle, the maximum elevation, that is, 15 centimeters, can be divided according to the preset accuracy threshold. The preset accuracy threshold can be selected as 1 cm, so 15 centimeters is divided into 1 cm, and the corresponding , the total number of space cubes contained in the space voxel is 2×15=30. For example, in order to facilitate data processing, the length and width of the space cube correspond to the same space size as the length and width of the unit grid, such as 0.01 meters × 0.01 meters. Correspondingly, the height of the space cube corresponds to the real space size Meters, so the real space size of each stereo frame is 0.01×0.01×0.005, in meters.

[0094] As can be seen from the above, this embodiment sets the parameters of the grid area according to the laser radar parameters of the vehicle, which is conducive to improving the recognition accuracy of road obstacles. The spatial voxel parameters are set by the maximum elevation of the target object. Since the spatial voxels are used to calculate the elevation information of the obstacle, the obstacle recognition accuracy can be further improved.

[0095] The above embodiment does not limit how to obtain grid elevation information. Based on the above embodiment, the present invention also provides an exemplary implementation method, which may include the following contents:

[0096] According to the inclination data of the horizontal plane of the road area to be detected and the vehicle coordinate system and the position information of each grid, the elevation difference between the road surface where the spatial voxels of each grid in the bird's-eye view grid area are located and the horizontal plane of the vehicle coordinate system is determined.

[0097] Exemplarily, the road area to be detected is divided into multiple types of road areas according to the inclination of the road surface corresponding to each grid, and the inclination of the same type of road area meets the preset same similar inclination conditions; based on the inclination data of each road area and the horizontal plane of the vehicle coordinate system and the position information of each grid, the elevation difference between the road surface where the spatial voxels of each grid in the bird's-eye view grid area are located and the horizontal plane of the vehicle coordinate system is determined. Among them, the point cloud data in each grid is plane fitting processed to obtain the corresponding three-dimensional space plane equation; the angle between the normal vector of the three-dimensional space plane equation of each grid and the normal vector of the horizontal plane of the vehicle coordinate system is used as the plane angle between the road area of ​​the corresponding grid and the horizontal plane of the vehicle coordinate system. Figure 5 As shown in the figure, the coordinate system vertical distance H between the origin of the laser radar coordinate system and the road surface is determined according to the laser radar installation position and the vehicle result parameters, and the coordinate system vertical distance H and the plane angle , determine the position information of the intersection between the horizontal coordinate axis of the vehicle coordinate system and the road surface corresponding to each grid . The road area division process may include: fitting the three-dimensional space plane equation of the corresponding grid area according to the three-dimensional position information of the point cloud data inside each grid; determining the average normal vector in each grid based on the three-dimensional space plane equation of each grid; and segmenting the road area to be detected according to the spatial position correspondence between the bird's-eye view grid area and the road area to be detected according to the clustering result of the average normal vector of each grid, so as to obtain multiple types of road areas. For each grid, determine the position information of the intersection of the horizontal axis of the vehicle coordinate system and the road surface corresponding to the current grid on the horizontal axis; based on the plane angle, calculate the elevation difference between the road surface where the spatial voxels of each grid in the bird's-eye view grid area are located and the horizontal plane of the vehicle coordinate system according to the position information of each grid on the horizontal axis and the position information of the corresponding intersection on the horizontal axis. As an efficient calculation method, the elevation calculation relationship can be stored locally in advance, and the elevation information of the spatial voxels corresponding to each grid (i, j) corresponding to the section of the road surface can be calculated by calling the elevation calculation relationship. , the elevation calculation formula can be expressed as:

[0098] ;

[0099] Where i and j represent the grid in the i-th row and j-th column in the bird's-eye view grid area, such as (2, 2) Figure 2 The grid in the second row and second column of the grid area in .

[0100] As can be seen from the above, this embodiment divides the road surface according to the inclination angle, and determines the road surface inclination angle in a simple manner, thereby effectively improving the determination accuracy of the elevation information of the obstacle.

[0101] In the above embodiment, there is no limitation on how to perform step S101. In this embodiment, an exemplary implementation method for obtaining the features of the point cloud data and image data of the road surface area to be detected in the target perspective after rasterization is provided, which may include the following contents:

[0102] When an obstacle is detected in the area ahead of the vehicle, the position of the obstacle is monitored during the vehicle's forward movement; when an obstacle enters the road surface area to be detected, the road surface area to be detected in front of the current vehicle is rasterized to generate a bird's-eye view grid area including multiple grids, each grid having spatial voxels in a horizontal direction perpendicular to the road surface; point cloud data and image data of the bird's-eye view grid area are obtained, and the point cloud data and image data are converted to a bird's-eye view perspective to obtain point cloud features and image features of the corresponding perspective.

[0103] In the present invention, considering that the horizontal / vertical angular resolution of the existing mainstream laser radar is better than 0.06°, the laser radar spatial perception capability increases with the closer the target distance. Calculated at an angular resolution of 0.06°, the minimum spatial perception size is 5.2 cm at a distance of 50 meters from the radar; the minimum spatial perception capability is 2.0 cm at a distance of 20 meters from the radar; the minimum spatial perception capability is 1 cm at a distance of 10 meters from the radar; and the minimum spatial perception capability is 0.5 cm at a distance of 5 meters from the radar. If there are obstacles in front of the vehicle, such as low raised obstacles and potholes, as the vehicle gradually approaches the vehicle during driving, the point cloud data information collected by the sensor will be richer. Since the near-field verification is more complicated and requires more computing resources, this step can first perform initial identification through the far-field area. If no suspected low raised obstacles / potholes are found in the far-field area, it is not necessary to start the near-field verification calculation process, thereby effectively reducing unnecessary computing burdens. Based on this, this step divides the area in front of the vehicle according to the distance between the vehicle and the preset distance threshold, and divides it into the area in front of the vehicle and the road area to be detected. The maximum distance between the road area to be detected and the vehicle in the direction of vehicle travel is less than or equal to the minimum distance between the area in front of the vehicle and the vehicle. Among them, the distance threshold can be preset according to the current number of laser radar lines or resolution. The lower the number of lines or resolution, the smaller the distance threshold. That is, the area in front of the vehicle is the far-field area, and the road area to be detected is the near-field area. Taking the distance threshold of 20m as an example, the area 20m in front of the vehicle is defined as the area in front of the vehicle. The size of the area in front of the vehicle is: 80m×20m, and the near-field area within 20m in front of the vehicle is the road area to be detected. The size of the road area to be detected is 20m×20m.

[0104] In this embodiment, the target perspective is the bev (Bird's Eye View) perspective. For the point cloud data of the laser radar, any point cloud target detection algorithm can be used, such as Pointpillars (algorithm name), Second (algorithm name), Centerpoint (algorithm name) feature extraction module and Neck (module name) module to obtain the features of the point cloud under the Bev perspective. Similarly, the network configuration adopts Figure 2 The bev grid and spatial voxels shown in the figure, the feature extraction module is used to extract the basic features of the image and generate feature maps of different scales. The Neck module helps the network to better detect targets of different scales by fusing features from different levels. It can also improve the feature representation ability, especially for the detection of small targets and complex scenes. In addition, the Neck module usually performs downsampling and upsampling operations on the feature map to better capture information of different scales. For image data, you can first use the commonly used image feature backbone network, such as resnet50 (network model name), to extract the two-dimensional features of the image, and then use the LSS algorithm or cross-attention method based on the calibration parameters of the camera, lidar, etc. to convert the two-dimensional features of the image into the features of the image in bev. Similarly, the network configuration adopts Figure 2 The bev grid and space voxels shown.

[0105] As can be seen from the above, this embodiment only identifies suspected low obstacles / potholes in the far field, obtains the coordinates of each suspected obstacle in the global coordinate system (that is, the world coordinate system), and sets the ID information for subsequent continuous interpretation. Then, when the low obstacle / pothole area gradually enters the near field area, each low obstacle / pothole area is identified and its three-dimensional shape data is obtained. When a suspected low obstacle / pothole area is found in the far field, the near field verification work begins. It not only effectively reduces unnecessary computing burden and speeds up the efficiency of determining three-dimensional shape data. Moreover, through multiple verifications, the obstacle recognition accuracy can also be improved.

[0106] The above embodiment does not limit how to obtain the initial three-dimensional shape information. Based on the above embodiment, the present invention also provides an exemplary implementation method for obtaining the three-dimensional shape information of the obstacle by fusing features, which may include the following contents:

[0107] Pre-build and train a 3D target detection network model; input the fused features into the 3D target detection network model to obtain the location information, category information and instance segmentation results of the obstacle; based on the instance segmentation results and the point cloud data in the obstacle area, determine the geometric description data of the obstacle through the 3D surface fitting results; use the location information, category information and geometric description data of the obstacle as the initial 3D shape information of the obstacle.

[0108] In this embodiment, the three-dimensional target detection network model can be trained based on any deep learning three-dimensional target detection algorithm. It may include a detection task head, a classification task head and an instance segmentation task head. The detection task head is used to identify obstacles and output obstacle location information; the classification task head is used to identify obstacle category information; the instance segmentation task head is used to segment obstacles on the target plane to obtain instance segmentation results. The point cloud features and image features under the target perspective are fused by convolution using a decoder-encoder or decoder mechanism or direct convolution. For fused features, features are first extracted through the backbone network and the neck module, and then the detection task head is used to obtain the location information of low obstacles / pothole areas; the classification task head is used to obtain obstacle category information, such as low obstacles, pits, cracks, road bumps, etc. Finally, the instance segmentation task head is used to obtain the segmentation results of obstacles such as low obstacles and pothole areas on the horizontal plane. Based on the segmentation results and its internal point cloud data, the geometric morphological description data of the obstacles can be obtained through a three-dimensional surface fitting method, such as Figure 8 As shown, the geometric description data can be long ,Width , the angle with the lane line , Figure 8 is a top view, is the x-axis of the vehicle coordinate system, is the y-axis of the vehicle coordinate system.

[0109] As can be seen from the above, this embodiment uses the excellent learning ability of the three-dimensional target detection network model to calculate the three-dimensional shape data of the obstacle, which is useful for improving the determination accuracy of the three-dimensional shape data of the obstacle.

[0110] In the field of computer vision, in order to solve the problems existing in monocular depth estimation, related technologies use a network structure to predict elevation information to replace monocular depth estimation, such as BEVheight (a visual three-dimensional object detection framework with height information designed based on roadside scenes) and RoadBEV (reconstruction of road surface in bird's-eye view). However, the error in predicted elevation of this method can reach about 2 cm, and this error makes it impossible to accurately judge low raised obstacles and potholes. The reason is that the accuracy of the annotation data, especially the accuracy of elevation information annotation, is difficult to meet. Accurate annotation of pixel-level elevation information is very time-consuming and the effect is difficult to control. Therefore, this embodiment provides a method for accurately obtaining elevation information guided by dense point clouds, which may include the following:

[0111] The spatial voxels of each grid in the rasterized road surface area to be detected and the corresponding initial weight values ​​are obtained; the elevation information of the grid corresponding to the current frame point cloud data is obtained as the new elevation information, and the weight values ​​of the spatial voxels at the corresponding positions are updated using the new elevation information.

[0112] In this embodiment, if Figure 6 As shown in the figure, there are multiple areas in the road area to be detected, that is, grid area types, where the weight distribution of different grids (i, j) in a region is different due to their elevation. The height varies with the is the elevation information of each grid (i, j) in the area. The height of the real road surface corresponding to each grid may be different, so the weight distribution of each grid will also be different. The grid area type includes but is not limited to the position of the low obstacle / pothole area identified in the previous frame in the grid area at the current moment, the position of the low obstacle / pothole area identified in the current frame in the grid area at the current moment, and the position of the low obstacle / pothole area identified in the far field in the grid area at the current moment. In order to further improve the calculation accuracy of the elevation information, this embodiment determines different weights according to different grid area types. For example, each grid area type in the rasterized road surface area to be detected can be obtained first; when the current grid belongs to the first type of obstacle area identified in the previous frame, if the current grid does not belong to the radar point cloud projection area, then based on the spatial voxel coordinate position, elevation information and coordinate position of each grid in the first type of obstacle area, the weight of the spatial voxel of the current grid is determined; if the current grid belongs to the radar point projection area, then based on the spatial voxel coordinate position of each grid in the first type of obstacle area, the elevation information and the coordinate position of the current grid, the weight of the spatial voxel of the current grid is determined. If the current grid belongs to the radar point projection area, then based on the spatial voxel coordinate position of each grid in the first type of obstacle area The weight of the spatial voxel of the current grid is determined based on the voxel coordinate position, elevation information, elevation information of the radar point projection area and the coordinate position of the current grid; when the current grid belongs to the second type of obstacle area identified by the current frame, if the current grid does not belong to the radar point cloud projection area, the weight of the spatial voxel of the current grid is determined based on the spatial voxel coordinate position of each grid in the second type of obstacle area; if the current grid belongs to the radar point projection area, the weight of the spatial voxel of the current grid is determined based on the spatial voxel coordinate position of each grid in the second type of obstacle area and the elevation information of the radar point projection area.

[0113] For example, if Figure 7 As shown, the coordinate range of the spatial voxel is -30~30. For different grid areas, the weight of the spatial voxel is calculated as follows:

[0114] For the position of the low obstacle / pothole area identified in the previous frame in the bird's-eye view grid area at the current moment, the corresponding spatial voxels are divided into two categories: the first category of grid areas is the corresponding bird's-eye view grid without radar point cloud projection, and the second category of grid areas is the corresponding bird's-eye view grid with radar point cloud projection. At this moment, the elevation information of all the grids contained in it is calculated , the elevation information of the radar point cloud projection is , the weight calculation of the first type of grid area can be realized by the following relationship (1), and the weight calculation of the second type of grid area can be realized by the following relationship (2):

[0115] ; (1)

[0116] . (2)

[0117] For the position of the low obstacle / pothole area identified in the current frame in the bird's-eye view grid area at the current moment, the corresponding spatial voxels are also divided into two categories. The third type of grid area has no radar point cloud projection on its corresponding bird's-eye view grid, and the fourth type of grid area has a radar point cloud projection on its corresponding bird's-eye view grid. The spatial voxel also has a radar point cloud projection in the bird's-eye view grid. Correspondingly, the elevation information of the radar point cloud projection is , the weight calculation of the third type of grid area can be realized by the following relationship (3), and the weight calculation of the fourth type of grid area can be realized by the following relationship (4):

[0118] ; (3)

[0119] . (4)

[0120] For the position of the low obstacle / pothole area identified in the far field in the bird's-eye view grid area at the current moment, the corresponding spatial voxels are also divided into two categories. The fifth type of grid area has no radar point cloud projection on its corresponding bird's-eye view grid, and the sixth type of grid area has a radar point cloud projection on its corresponding bird's-eye view grid. The spatial voxel also has a radar point cloud projection in the bird's-eye view grid. Correspondingly, the elevation information of the radar point cloud projection is , the weight calculation of the fifth type of grid area can be realized by the following relationship (5), and the weight calculation of the sixth type of grid area can be realized by the following relationship (6):

[0121] ; (5)

[0122] . (6)

[0123] For example, as a simple weight setting method, a one-dimensional Gaussian function relationship can be used to generate a weight calculation relationship based on the preset Gaussian curve control parameter value, the center position coordinates determined by different grid area types, and the grid space voxel position information as variables, so as to determine the weight value of each grid space voxel by calling the weight calculation relationship. Can be 1.0, Can be 1.0, the weight of the spatial voxel can be correspondingly a one-dimensional Gaussian curve express, is the center position of the curve, g(x) represents the Gaussian curve, and in this embodiment, x represents the coordinate value of the spatial voxel.

[0124] When the weight calculation method of different grid areas is determined, the elevation information of the grid in the obstacle area is obtained through the elevation classification network. Among them, the elevation classification network is a pre-trained elevation classification network, which is used to convert the depth estimation problem into a classification problem, and the elevation classification network determines the loss function based on the obstacle type, elevation true value encoding information, obstacle area mask information, point cloud features and image features of the obstacle area grid, and spatial voxel weights of the obstacle area grid.

[0125] In this embodiment, based on the LSS method, the depth estimation problem is converted into a classification problem, and the elevation information of each bird's-eye view grid corresponding to the obstacle inside the road area to be detected can be obtained through the elevation classification network. In order to further improve the accuracy of the elevation information, this embodiment also provides a corresponding loss function. As an efficient implementation method, the loss function can be pre-stored. The loss function is determined based on the obstacle type, elevation true value coding information, obstacle area mask information, point cloud features and image features of the obstacle area grid, and the spatial voxel weight of the obstacle area grid. The elevation true value coding information is processed using one-hot coding, which can be expressed in the following mathematical form:

[0126] ;

[0127] in, , is the number of types of obstacles identified, is a one-hot GT mask (representing elevation truth encoding information), which is a mask generated using pre-labeled elevation truth values. is the mask information of the obstacle, For the A bird's-eye view grid is used to obtain point cloud features and image features. The loss function is based on common knowledge in the field of deep learning. Since the base does not affect the model training process, it is usually ignored.

[0128] After the road area to be detected is rasterized, for example, if the number of grids is 2000×2000 and the voxel is set to 60 elevation estimates, if the calculation is performed point by point, the amount of calculation is huge. Usually, obstacles only occupy part of the grid, and for non-low obstacles and potholes, it is not necessary to calculate their elevation information. In order to reduce the amount of calculation, the present invention can also first determine the grids contained in the obstacle, and the calculation of the elevation information of the grid can include the following:

[0129] According to the position information of the obstacle identified in the previous frame at the current moment of the bird's-eye view grid area, and the position information of the obstacle identified in the current frame at the current moment of the bird's-eye view grid area, the grid included in the obstacle area of ​​the current frame is determined; the elevation information of the grid included in the obstacle area of ​​the current frame is obtained, and the spatial voxel weights of the grid included in the obstacle area of ​​the current frame are updated.

[0130] This embodiment only retains the low obstacles / potholes identified in the previous frame, the low obstacles / potholes identified in the current frame, and the suspected low obstacles / potholes that enter the near field area identified in the far field. In order to further effectively reduce the amount of calculation, this embodiment also generates mask information based on the grid occupied by the obstacle. Exemplarily, the method for determining the grid contained in the obstacle area of ​​the current frame is: according to the obstacle recognition results corresponding to the previous frame and the current frame, determine the grid in the corresponding obstacle area; obtain the position information of the bird's-eye view grid area of ​​the obstacle detected in the area ahead of the vehicle at the current moment; the minimum distance between the area ahead of the vehicle and the vehicle along the vehicle's driving direction is greater than or equal to the maximum distance between the road surface area to be detected and the vehicle; according to the position change of the vehicle in the local world coordinate system in the adjacent frame time period, update the position information of the grid in the obstacle area of ​​the previous frame as the position information of the bird's-eye view grid area of ​​the obstacle identified in the previous frame at the current moment; if the current point position is the obstacle identified in the previous frame If the position of the current point is the position information of the obstacle in the bird's-eye view grid area at the current moment, or the position information of the obstacle identified in the current frame in the bird's-eye view grid area at the current moment, or the position information of the obstacle identified in the front driving area in the bird's-eye view grid area at the current moment, a first preset value is set for the current point as a mask value; if the current point position is not the position information of the obstacle identified in the previous frame in the bird's-eye view grid area at the current moment, or the position information of the obstacle identified in the current frame in the bird's-eye view grid area at the current moment, or the position information of the obstacle identified in the front driving area in the bird's-eye view grid area at the current moment, a second preset value is set for the current point as a mask value; according to the mask value of each point in the road surface area to be detected, the obstacle area mask information is generated to identify the grid contained in the obstacle area of ​​the current frame.

[0131] In this embodiment, the mask information is automatically generated based on the radar projection point, the position of the low obstacle / pothole area identified in the previous frame at the current grid area, the position of the low obstacle / pothole area currently identified at the current grid area, and the position of the low obstacle / pothole area identified in the far field at the current grid area, without manual setting. Figure 6As shown in the figure, the first type of area in the grid is the projection point of the laser radar point cloud on the road surface, that is, the intersection of the laser radar beam and the ground. If there are multiple laser radar point cloud projection points in the same grid, the z coordinate takes their average value; the second type of area in the figure is the low obstacle / pothole area identified in the previous frame. As the vehicle continues to move forward, the elevation information of the identified area is gradually updated through subsequent calculations; as the vehicle moves forward, the distance of the low obstacle / pothole area identified in the previous frame relative to the vehicle changes, and it is necessary to update the corresponding position of each low obstacle / pothole area identified in the previous frame in the grid area according to the change in the position of the vehicle in the local world coordinate system. For example, the grid contained in the obstacle at the previous moment, that is, at time t-1 Location is At the current time t, the vehicle has traveled 0.2 meters straight forward. If the spatial distance corresponding to a grid is 0.01 meters, then The coordinates need to be updated according to the updated x=x-vehicle travel distance*grid corresponding space size. The updated grid can be expressed as , Corresponding to the x-axis of the vehicle coordinate system, Corresponding to the y-axis of the vehicle coordinate system. The third type of area in the figure is a suspected obstacle detected in the far field. As the vehicle continues to move forward and gradually enters the grid area, it is necessary to verify whether it is a real obstacle. If it is not a real obstacle, the three-dimensional shape information of this type of area will not be calculated later. If it is a low obstacle or a pothole area, its accurate three-dimensional shape data will be further calculated. The fourth type of area in the figure is the obstacle area identified in the current frame.

[0132] For example, as a more efficient method, a pre-stored mask relationship can be called to generate corresponding mask information. , the mask relation can be expressed as:

[0133] ;

[0134] Among them, i and j represent a grid, and the set D includes: the position of the obstacle recognized in the previous frame in the grid area at the current moment; the position of the obstacle recognized currently in the grid area at the current moment; the position of the obstacle recognized in the far field in the grid area at the current moment.

[0135] It can be seen from the above that this embodiment generates obstacle mask information by utilizing the previous frame results, multimodal features, and far-field recognition results, which can not only effectively reduce the amount of calculation, but also help improve the accuracy of the three-dimensional shape data of the obstacle.

[0136] In order to make the technical solution of the present invention more clear to those skilled in the art, the present invention also provides an exemplary implementation mode, as shown in FIG. Fig. 9 As shown, it may include the following:

[0137] This embodiment can pre-construct a three-dimensional shape data generation model, and the user inputs the following data into the three-dimensional shape data generation model: near-field dense point cloud or single-frame point cloud data after distortion is eliminated, forward camera image, local world coordinate system and local world point cloud, recognition result of the previous frame, far-field detection result, and sensor calibration parameters. Among them, the sensor calibration parameters include calibration parameters of camera and lidar, lidar and inertial measurement unit, inertial measurement unit and odometer. As the vehicle continues to move forward, the point cloud data at different times are spliced ​​along the time axis to form a local world point cloud. The corresponding point cloud bev features are obtained by establishing the local world point cloud map and dense point cloud data. Among them, for mechanical lidar point cloud data, within a frame of data acquisition cycle, each frame of point cloud data has motion distortion due to vehicle movement. The distortion calibration method can adopt any method, such as obtaining the kinematic data of the vehicle, estimating the vehicle's movement distance and direction within 100 milliseconds, assuming uniform linear motion, and based on this, calibrate the point cloud xyz coordinates in combination with the timestamp of each point.

[0138] The 3D shape data generation model converts point cloud data into point cloud features under the target perspective such as bev, which can also be defined as point cloud bev features. The image data is converted into image features under the target perspective such as bev in combination with the sensor calibration parameters, which can also be defined as image bev features. Then the point cloud bev features and the image bev features are fused through the encoder to obtain the fusion features under bev, and then the obstacle positioning recognition results, two-dimensional recognition results and obstacle category results are obtained through the decoder. The three-dimensional recognition result is obtained based on the two-dimensional recognition result and the new elevation information of each spatial voxel. The obstacle mask information is jointly generated based on the recognition result of the previous frame, the far-field detection result and the current frame recognition result. The obstacle recognition result of the road area to be detected is obtained through the obstacle mask information, the three-dimensional recognition result and the point cloud bev feature. The final output of the 3D shape data generation model is: the three-dimensional shape information, positioning information and category information of the obstacle in front of the vehicle.

[0139] It can be seen from the above that this embodiment can continuously improve the accuracy of elevation information and instance segmentation information through iterative calculation of the three-dimensional shape data generation model, thereby continuously improving the accuracy of the three-dimensional shape information of obstacles in the road area to be detected.

[0140] It should be noted that there is no strict order of execution between the steps in the present invention. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 1 and Fig. 9 This is just a schematic and does not mean that this is the only execution order.

[0141] The present invention also provides a corresponding device for the method for determining the three-dimensional shape information of road obstacles, which further makes the method more practical. Among them, the device can be described from the perspective of functional modules and hardware. The following is an introduction to the device for determining the three-dimensional shape information of road obstacles provided by the present invention. The device is used to implement the method for determining the three-dimensional shape information of road obstacles provided by the present invention. In this embodiment, the device for determining the three-dimensional shape information of road obstacles can include or be divided into one or more program modules, which are stored in a storage medium and executed by one or more processors to complete the method for determining the three-dimensional shape information of road obstacles disclosed in the first embodiment. The program module referred to in this embodiment refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of the device for determining the three-dimensional shape information of road obstacles in the storage medium than the program itself. The following description will specifically introduce the functions of each program module of this embodiment. The device for determining the three-dimensional shape information of road obstacles described below and the method for determining the three-dimensional shape information of road obstacles described above can be referred to each other.

[0142] From the perspective of functional modules, see Fig.10 , Fig.10 The structure diagram of the device for determining the three-dimensional shape information of road obstacles provided in this embodiment under a specific implementation manner, the device may include:

[0143] The multimodal feature acquisition module 101 is used to acquire the features of the point cloud data and image data of the rasterized road area to be detected under the target perspective; wherein the rasterized road area to be detected includes multiple grids, and each grid is provided with spatial voxels in a horizontal direction perpendicular to the road surface.

[0144] The initial three-dimensional shape information determination module 102 is used to fuse the point cloud features and the image features, and perform target detection processing on the fused features to obtain the initial three-dimensional shape information of the obstacles in the road area to be detected.

[0145] The precision update module 103 is used to update the spatial voxel weight at the current moment according to the new elevation information of the grid corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement, and to update the elevation information and geometric data of the grid in the obstacle area based on the updated spatial voxel weight, so as to continuously update the initial three-dimensional shape information.

[0146] Exemplarily, in some implementations of this embodiment, the multimodal feature acquisition module 101 can also be used to: when an obstacle is detected in the area ahead of the vehicle, monitor the position of the obstacle during the vehicle's forward movement; when an obstacle enters the road surface area to be detected, rasterize the road surface area to be detected in front of the current vehicle to generate a bird's-eye view grid area including multiple grids, each grid having spatial voxels in a horizontal direction perpendicular to the road surface; obtain point cloud data and image data of the bird's-eye view grid area, and convert the point cloud data and image data to a bird's-eye view perspective to obtain point cloud features and image features of the corresponding perspective. Wherein, the maximum distance between the road surface area to be detected and the vehicle along the vehicle's driving direction is less than or equal to the minimum distance between the area ahead of the vehicle and the vehicle.

[0147] Exemplarily, in some other implementations of the present embodiment, the above-mentioned initial three-dimensional shape information determination module 102 can also be used to: pre-build and train a three-dimensional target detection network model; the three-dimensional target detection network model includes a detection task head, a classification task head and an instance segmentation task head, the detection task head is used to identify obstacles and output obstacle position information; the classification task head is used to identify obstacle category information; the instance segmentation task head is used to segment obstacles on the target plane to obtain instance segmentation results; the fusion features are input into the three-dimensional target detection network model to obtain the obstacle position information, category information and instance segmentation results; based on the instance segmentation results and the point cloud data in the obstacle area, the geometric morphology description data of the obstacle is determined through the three-dimensional surface fitting results; the obstacle position information, category information and geometric morphology description data are used as the initial three-dimensional shape information of the obstacle.

[0148] Exemplarily, in some other implementations of this embodiment, the above-mentioned device may also include a data conversion module, which can be used to: based on the transformation relationship between the vehicle coordinate system and the local world coordinate system, convert the three-dimensional shape information of obstacles in the road area to be detected into the local world coordinate system; wherein the local world coordinate system is the coordinate system of the road surface on which the vehicle is traveling.

[0149] Exemplarily, in some other implementations of the present embodiment, the above-mentioned device may also include a transformation relationship determination module, which can be used to: obtain point cloud data and local map point cloud at the current moment; determine the transformation relationship between the vehicle coordinate system and the local world coordinate system by performing point cloud matching on the point cloud data and the local map point cloud; wherein the local map point cloud is data generated by splicing point cloud data at different moments along the time axis.

[0150] Exemplarily, in some other implementations of the present embodiment, the above-mentioned precision update module 103 can also be used to: obtain the spatial voxels of each grid in the rasterized road surface area to be inspected and the corresponding initial weight values; obtain the elevation information of the grid corresponding to the current frame point cloud data as new elevation information, and use the new elevation information to update the weight values ​​of the spatial voxels at the corresponding positions.

[0151] Exemplarily, in some other implementations of this embodiment, the precision updating module 103 may also be used to: obtain the type of each grid area in the gridded road surface area to be detected; when the current grid belongs to the first type of obstacle area identified in the previous frame, if the current grid does not belong to the radar point cloud projection area, then based on the spatial voxel coordinate position of each grid in the first type of obstacle area, the elevation information and the coordinate position of the current grid, determine the weight of the spatial voxel of the current grid; if the current grid belongs to the radar point projection area, then based on the spatial voxel coordinate position of each grid in the first type of obstacle area, the elevation information and the coordinate position of the current grid, determine the weight of the spatial voxel of the current grid; if the current grid belongs to the radar point projection area, then based on the spatial voxel coordinate position of each grid in the first type of obstacle area, determine the weight of the spatial voxel of the current grid; The weight of the spatial voxel of the current grid is determined based on the location, elevation information, elevation information of the radar point projection area and the coordinate position of the current grid; when the current grid belongs to the second type of obstacle area identified by the current frame, if the current grid does not belong to the radar point cloud projection area, the weight of the spatial voxel of the current grid is determined based on the spatial voxel coordinate position of each grid in the second type of obstacle area; if the current grid belongs to the radar point projection area, the weight of the spatial voxel of the current grid is determined based on the spatial voxel coordinate position of each grid in the second type of obstacle area and the elevation information of the radar point projection area.

[0152] Exemplarily, in some other implementations of the present embodiment, the above-mentioned precision update module 103 can be further used to: based on a one-dimensional Gaussian function relationship, generate a weight calculation relationship according to preset Gaussian curve control parameter values, center position coordinates determined by different grid area types, and grid space voxel position information as variables, so as to determine the weight value of each grid space voxel by calling the weight calculation relationship.

[0153] Exemplarily, in some other implementations of the present embodiment, the above-mentioned precision update module 103 can also be used for: pre-training an elevation classification network, the elevation classification network is used to convert the depth estimation problem into a classification problem, and the elevation classification network is based on the obstacle type, elevation true value coding information, obstacle area mask information, point cloud features and image features of the obstacle area grid, and spatial voxel weights of the obstacle area grid to determine the loss function; the elevation information of the grid in the obstacle area is obtained through the elevation classification network.

[0154] Exemplarily, in some other implementations of this embodiment, the above-mentioned precision update module 103 can also be used to: determine the grid included in the obstacle area of ​​the current frame according to the position information of the obstacle identified in the previous frame at the current moment in the bird's-eye view grid area, and the position information of the obstacle identified in the current frame at the current moment in the bird's-eye view grid area; obtain the elevation information of the grid included in the obstacle area of ​​the current frame, and update the spatial voxel weights of the grid included in the obstacle area of ​​the current frame.

[0155] As an exemplary implementation of the above embodiment, the above precision update module 103 can also be further used to: determine the grid in the corresponding obstacle area according to the obstacle recognition results corresponding to the previous frame and the current frame; obtain the position information of the bird's-eye view grid area of ​​the obstacle detected in the area ahead of the vehicle at the current moment; the minimum distance between the area ahead of the vehicle and the vehicle along the vehicle's driving direction is greater than or equal to the maximum distance between the road surface area to be detected and the vehicle; update the position information of the grid in the obstacle area of ​​the previous frame according to the position change of the vehicle in the local world coordinate system in the adjacent frame time period, as the position information of the bird's-eye view grid area of ​​the obstacle identified in the previous frame at the current moment; if the current point position is the bird's-eye view grid area of ​​the obstacle identified in the previous frame at the current moment If the position information of the grid area or the position information of the bird's-eye view grid area of ​​the obstacle identified in the current frame at the current moment or the position information of the bird's-eye view grid area of ​​the obstacle identified in the front driving area at the current moment is present, a first preset value is set for the current point as a mask value; if the current point position is not the position information of the bird's-eye view grid area of ​​the obstacle identified in the previous frame at the current moment or the position information of the bird's-eye view grid area of ​​the obstacle identified in the current frame at the current moment or the position information of the bird's-eye view grid area of ​​the obstacle identified in the front driving area at the current moment, a second preset value is set for the current point as a mask value; according to the mask value of each point in the road surface area to be detected, the obstacle area mask information is generated to identify the grid contained in the obstacle area of ​​the current frame.

[0156] The device for determining the three-dimensional shape information of road obstacles mentioned above is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. Fig.11 The following is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention in one implementation manner. Fig.11 As shown, the electronic device includes a memory 110 for storing a computer program; a processor 111 for implementing the steps of the method for determining three-dimensional shape information of road obstacles mentioned in any of the above embodiments when executing the computer program.

[0157] Among them, the processor 111 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 111 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 111 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 111 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 111 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 111 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0158] The memory 110 may include one or more computer non-volatile storage media, which may be non-transitory. The memory 110 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the memory 110 may be an internal storage unit of an electronic device, such as a hard disk of a server. In other embodiments, the memory 110 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 110 may also include both an internal storage unit of an electronic device and an external storage device. The memory 110 can not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program in the process of executing the method for determining the three-dimensional shape information of road obstacles, but also can be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 110 is used to store at least the following computer program 1101, wherein after the computer program is loaded and executed by the processor 111, the relevant steps of the method for determining the three-dimensional shape information of the road obstacle disclosed in any of the above embodiments can be implemented. In addition, the resources stored in the memory 110 may also include an operating system 1102 and data 1103, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 1102 may include Windows, Unix, Linux, etc. The data 1103 may include, but is not limited to, data corresponding to the result of determining the three-dimensional shape information of the road obstacle, etc.

[0159] In some embodiments, the electronic device may further include a display screen 112, an input / output interface 113, a communication interface 114 or a network interface, a power supply 115 and a communication bus 116. Among them, the display screen 112 and the input / output interface 113, such as a keyboard, belong to the user interface, and the exemplary user interface may also include a standard wired interface, a wireless interface, etc. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 114 may exemplarily include a wired interface and / or a wireless interface, such as a WI-FI interface, a Bluetooth interface, etc., which is generally used to establish a communication connection between the electronic device and other electronic devices. The communication bus 116 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0160] Those skilled in the art will understand that Fig.11 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, for example, it may also include a sensor 117 to implement various functions.

[0161] It is understandable that if the method for determining the three-dimensional shape information of road obstacles in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes but is not limited to: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes. Based on this, the present invention also provides a non-volatile storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for determining the three-dimensional shape information of road obstacles recorded in any of the above embodiments are executed.

[0162] It is understandable that if the method for determining the three-dimensional shape information of road obstacles in the above-mentioned embodiments is implemented in the form of a software functional unit and sold or used as an independent product, the computer software product does not need to be stored in a physical storage medium. For example, it can be directly transmitted to a computer or other device with information processing capabilities through a wired network or a wireless network to perform all or part of the steps of the methods of various embodiments of the present invention. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product. Based on this, the present invention also provides a computer program product, which stores a computer program. When the computer program is executed by a processor, the steps of the method for determining the three-dimensional shape information of road obstacles recorded in any of the above embodiments are performed.

[0163] Finally, the present invention also provides an unmanned vehicle that fully reuses its own sensors, such as cameras, lidars, odometers, and inertial measurement units, without the need to install other sensors. A computer program that implements the method for determining the three-dimensional shape information of road obstacles as recorded in any of the previous embodiments is built into the existing automatic driving perception system. The automatic driving perception system can be used to implement the steps of the method for determining the three-dimensional shape information of road obstacles as recorded in any of the previous embodiments when executing the computer program stored in the memory. The accuracy of identifying low raised obstacles / pothole areas is increased by a point cloud densification method. Based on multi-source sensor data, according to the characteristics of near-field data, through detection, identification and verification, the three-dimensional position and three-dimensional shape data of low obstacles / pothole areas relative to the unmanned vehicle can be accurately identified, such as height, depth, and size. In this way, the unmanned vehicle can accurately identify low raised obstacles such as speed bumps and cracks, deep pits, shallow pits or puddles as early as possible in the automatic driving scenario. Fig.12 As shown, low obstacle / pothole information is integrated into autonomous driving planning and control to enhance the vehicle’s obstacle avoidance capability, effectively improving the safety and comfort of the autonomous driving system.

[0164] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the hardware disclosed in the embodiments, including electronic devices, non-volatile storage media, computer program products and unmanned vehicles, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0165] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0166] The above is a detailed introduction to a method for determining three-dimensional morphological information of road obstacles, an electronic device, a non-volatile storage medium, a computer program product and an unmanned vehicle provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that based on the embodiments of the present invention, for ordinary technicians in this technical field, all other embodiments obtained without creative work are within the scope of protection of the present invention. Without departing from the principles of the present invention, the present invention may also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A method for determining three-dimensional shape information of road obstacles, characterized in that: include: Acquire the features of the point cloud data and image data of the road surface area to be detected in the target perspective after rasterization; wherein the road surface area to be detected in the rasterization includes a plurality of grids, and each grid is provided with a spatial voxel in a horizontal direction perpendicular to the road surface; The point cloud features and image features are fused, and the initial three-dimensional shape information of the obstacles in the road area to be detected is obtained by performing target detection processing on the fused features; According to the new grid elevation information corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement, the spatial voxel weight at the current moment is updated, and the elevation information and geometric morphology data of the grid in the obstacle area are updated based on the updated spatial voxel weight, so as to continuously update the initial three-dimensional shape information; The updating of the spatial voxel weight at the current moment according to the new grid elevation information corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement includes: Obtain the type of each grid area in the rasterized road surface area to be detected; When the current grid belongs to the first type of obstacle area identified in the previous frame, if the current grid does not belong to the radar point cloud projection area, the weight of the spatial voxel of the current grid is determined based on the spatial voxel coordinate position, elevation information of each grid in the first type of obstacle area and the coordinate position of the current grid; if the current grid belongs to the radar point cloud projection area, the weight of the spatial voxel of the current grid is determined based on the spatial voxel coordinate position, elevation information of each grid in the first type of obstacle area, the elevation information of the radar point cloud projection area and the coordinate position of the current grid; When the current grid belongs to the second type of obstacle area identified by the current frame, if the current grid does not belong to the radar point cloud projection area, the weight of the spatial voxel of the current grid is determined based on the spatial voxel coordinate position of each grid in the second type of obstacle area; if the current grid belongs to the radar point cloud projection area, the weight of the spatial voxel of the current grid is determined based on the spatial voxel coordinate position of each grid in the second type of obstacle area and the elevation information of the radar point cloud projection area.

2. The method for determining the three-dimensional shape information of road obstacles according to claim 1, characterized in that: The step of obtaining the features of the point cloud data and image data of the road area to be detected in the target viewing angle after rasterization processing includes: When an obstacle is detected in the area ahead of the vehicle, the position of the obstacle is monitored during the forward movement of the vehicle; When the obstacle enters the road surface area to be detected, the road surface area to be detected in front of the current vehicle is rasterized to generate a bird's-eye view grid area including a plurality of grids, each grid having spatial voxels in a horizontal direction perpendicular to the road surface; Acquire point cloud data and image data of the bird's-eye view grid area, and convert the point cloud data and image data to a bird's-eye view to obtain point cloud features and image features of the corresponding view; The maximum distance between the to-be-detected road surface area and the vehicle along the vehicle's driving direction is less than or equal to the minimum distance between the area in front of the vehicle and the vehicle.

3. The method for determining the three-dimensional shape information of a road obstacle according to claim 1, characterized in that: By performing target detection processing on the fused features, the initial three-dimensional shape information of the obstacles in the road area to be detected is obtained, including: Pre-constructing and training a three-dimensional target detection network model; the three-dimensional target detection network model includes a detection task head, a classification task head and an instance segmentation task head, the detection task head is used to identify obstacles and output obstacle position information; the classification task head is used to identify obstacle category information; the instance segmentation task head is used to segment obstacles on the target plane to obtain instance segmentation results; Inputting the fused features into the three-dimensional target detection network model to obtain the location information, category information and instance segmentation results of the obstacles; Based on the instance segmentation result and the point cloud data in the obstacle area, determining the geometric description data of the obstacle through the three-dimensional surface fitting result; The position information, category information and geometric description data of the obstacle are used as the initial three-dimensional shape information of the obstacle.

4. The method for determining the three-dimensional shape information of a road obstacle according to claim 1, characterized in that: After updating the elevation information and geometric data of the grid in the obstacle area based on the updated spatial voxel weight, the method further includes: Based on the transformation relationship between the vehicle coordinate system and the local world coordinate system, the three-dimensional shape information of the obstacle in the road area to be detected is converted into the local world coordinate system; The local world coordinate system is the coordinate system of the road surface on which the vehicle is traveling.

5. The method for determining the three-dimensional shape information of a road obstacle according to claim 4, characterized in that: Before the transformation relationship between the vehicle coordinate system and the local world coordinate system is described, the method further includes: Get the point cloud data and local map point cloud at the current moment; Determine the transformation relationship between the vehicle coordinate system and the local world coordinate system by performing point cloud matching on the point cloud data and the local map point cloud; The local map point cloud is data generated by splicing point cloud data at different times along the time axis.

6. The method for determining the three-dimensional shape information of a road obstacle according to claim 1, characterized in that: According to the new grid elevation information corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement, the spatial voxel weight at the current moment is updated, including: Obtaining the spatial voxels of each grid of the rasterized road surface area to be detected and the corresponding initial weight values; The elevation information of the grid corresponding to the point cloud data of the current frame is obtained as the new elevation information, and the weight value of the spatial voxel at the corresponding position is updated using the new elevation information.

7. The method for determining the three-dimensional shape information of a road obstacle according to claim 1, characterized in that: Before obtaining the types of each grid area in the gridded road surface area to be detected, the method further includes: Based on the one-dimensional Gaussian function relationship, a weight calculation relationship is generated according to the preset Gaussian curve control parameter value, the center position coordinates determined by different grid area types, and the grid space voxel position information as variables, so as to determine the weight value of each grid space voxel by calling the weight calculation relationship.

8. The method for determining the three-dimensional shape information of a road obstacle according to claim 1, characterized in that: The updating of the elevation information of the grid in the obstacle area based on the updated spatial voxel weight includes: Pre-training an elevation classification network, the elevation classification network is used to convert the depth estimation problem into a classification problem, and the elevation classification network determines a loss function based on obstacle type, elevation truth encoding information, obstacle area mask information, point cloud features and image features of the obstacle area grid, and spatial voxel weights of the obstacle area grid; The elevation information of the grid in the obstacle area is obtained through the elevation classification network.

9. The method for determining the three-dimensional shape information of a road obstacle according to any one of claims 1 to 8, characterized in that: According to the new grid elevation information corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement, the spatial voxel weight at the current moment is updated, including: Determine the grids included in the obstacle area of ​​the current frame according to the position information of the obstacle identified in the previous frame in the bird's-eye view grid area at the current moment and the position information of the obstacle identified in the current frame in the bird's-eye view grid area at the current moment; The elevation information of the grid included in the obstacle area of ​​the current frame is obtained, and the spatial voxel weights of the grid included in the obstacle area of ​​the current frame are updated.

10. The method for determining the three-dimensional shape information of a road obstacle according to claim 9, characterized in that: According to the position information of the obstacle identified in the previous frame in the bird's-eye view grid area at the current moment and the position information of the obstacle identified in the current frame in the bird's-eye view grid area at the current moment, the grids included in the obstacle area of ​​the current frame are determined, including: According to the obstacle recognition results corresponding to the previous frame and the current frame, the grid in the corresponding obstacle area is determined; Obtaining the position information of the obstacle detected in the area ahead of the vehicle at the current moment in the bird's-eye view grid area; the minimum distance between the area ahead of the vehicle and the vehicle along the vehicle's driving direction is greater than or equal to the maximum distance between the road surface area to be detected and the vehicle; According to the position change of the vehicle in the local world coordinate system in the adjacent frame time period, the position information of the grid in the obstacle area of ​​the previous frame is updated to serve as the position information of the grid area of ​​the bird's-eye view of the obstacle identified in the previous frame at the current moment; If the current point position is the position information of the obstacle identified in the previous frame at the current moment in the bird's-eye view grid area, or the position information of the obstacle identified in the current frame at the current moment in the bird's-eye view grid area, or the position information of the obstacle identified in the front driving area at the current moment in the bird's-eye view grid area, then a first preset value is set for the current point as a mask value; if the current point position is not the position information of the obstacle identified in the previous frame at the current moment in the bird's-eye view grid area, or the position information of the obstacle identified in the current frame at the current moment in the bird's-eye view grid area, or the position information of the obstacle identified in the front driving area at the current moment in the bird's-eye view grid area, then a second preset value is set for the current point as a mask value; Obstacle area mask information is generated according to the mask value of each point in the road area to be detected, so as to identify the grids included in the obstacle area of ​​the current frame.

11. An electronic device, characterized in that: It comprises a processor and a memory, wherein the processor is used to implement the steps of the method for determining three-dimensional shape information of road obstacles as claimed in any one of claims 1 to 10 when executing the computer program stored in the memory.

12. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for determining three-dimensional shape information of road obstacles as described in any one of claims 1 to 10.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the method for determining three-dimensional shape information of road obstacles described in any one of claims 1 to 10 are implemented.

14. An unmanned vehicle, characterized in that: It includes an autonomous driving perception system, which is used to implement the steps of the method for determining three-dimensional shape information of road obstacles as described in any one of claims 1 to 10 when executing a computer program stored in a memory.

Images