Cost map generation method, device, medium, product and unmanned vehicle

Abstract

The present invention discloses a cost map generation method, equipment, medium, product and unmanned vehicle, which are applied to the field of autonomous driving technology. The method includes constructing a blank three-dimensional cost map for a driving area to be detected based on rasterization processing and raster setting of spatial voxels. The three-dimensional spatial data of the first type of obstacles are obtained by performing target recognition on the image data and point cloud data of the driving area to be detected; based on the fusion characteristics of the image data and point cloud data under the target perspective, the spatial voxel weight at the current moment is updated according to the new grid elevation information corresponding to each frame of point cloud data continuously obtained during the vehicle's forward movement, and the three-dimensional morphological information of the second type of obstacles is determined based on the updated spatial voxel weight. The three-dimensional spatial data and three-dimensional morphological information are filled into the blank three-dimensional cost map to generate a three-dimensional cost map. The present invention can solve the problem that related technologies cannot accurately avoid obstacles, improve the richness of cost maps, and improve the obstacle avoidance ability of vehicles.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a cost map generation method, electronic equipment, non-volatile storage medium, computer program product and unmanned vehicle. Background Art

[0002] With the rapid development of artificial intelligence technology and Internet of Things technology, intelligent driving technology has also developed accordingly. The cost map is a grid map established after integrating multi-sensor information. It can provide rich environmental information for autonomous driving vehicles, enabling them to intelligently plan paths, avoid obstacles, and automatically generate adaptive decisions during the execution of driving tasks.

[0003] The cost map of the related technology is a two-dimensional plane information given from an omniscient perspective, such as Figure 1 As shown, when the intelligent vehicle is driving according to the cost map, it cannot accurately avoid obstacles and has poor obstacle avoidance ability, which causes the intelligent driving vehicle to be unable to complete the driving task efficiently and safely.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present invention provides a cost map generation method, electronic equipment, non-volatile storage medium, computer program product and unmanned vehicle. The cost map contains richer information and can effectively improve the obstacle avoidance capability of the intelligent driving vehicle.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In one aspect, the present invention provides a cost map generation method, comprising:

[0008] A blank three-dimensional cost map is constructed based on a driving area to be detected that is rasterized and each grid includes spatial voxels in a horizontal direction perpendicular to the road surface; three-dimensional spatial data of a first type of obstacle is obtained by performing target recognition processing on the image data and point cloud data of the driving area to be detected; based on the point-graph fusion features of the image data and point cloud data of the driving area to be detected under the target perspective, the spatial voxel weight at the current moment is updated according to the new elevation information of the grid corresponding to each frame of point cloud data continuously obtained during the vehicle's forward movement, and the three-dimensional shape information of a second type of obstacle is determined based on the updated spatial voxel weight; wherein the elevation information of the second type of obstacle is less than the elevation information of the first type of obstacle; the blank three-dimensional cost map is filled according to the three-dimensional spatial data and the three-dimensional shape information to obtain a three-dimensional cost map.

[0009] In a first exemplary embodiment, before filling the blank three-dimensional cost map according to the three-dimensional spatial data and the three-dimensional shape information to obtain the three-dimensional cost map, the method further includes:

[0010] According to the ground area determined by the image data and the calibration parameters between the laser radar and the camera, the ground point cloud data corresponding to the ground area is extracted from the point cloud data; according to the clustering result of the average normal vector of each grid in the ground area, the ground area is segmented to obtain a plurality of sub-ground areas; according to the point cloud data contained in each sub-ground area, the corresponding sub-ground area is plane fitting processed to determine the ground elevation information and ground inclination information of each sub-ground, so as to obtain the three-dimensional ground data as the ground of the three-dimensional cost map.

[0011] In a second exemplary embodiment, filling the blank three-dimensional cost map according to the three-dimensional spatial data and the three-dimensional shape information to obtain a three-dimensional cost map includes:

[0012] According to using the three-dimensional ground data as the three-dimensional ground layer of the three-dimensional cost map, the blank three-dimensional cost map is filled to obtain an initial three-dimensional cost map; the three-dimensional spatial data and the three-dimensional morphology information are added to the initial three-dimensional cost map to generate a three-dimensional cost map.

[0013] In a third exemplary embodiment, the first type of obstacles includes dynamic obstacles and static obstacles, and adding the three-dimensional spatial data and the three-dimensional shape information to the initial three-dimensional cost map includes:

[0014] The three-dimensional bounding box and posture information of the dynamic obstacle are added to the initial three-dimensional cost map; for each static obstacle, if the overlapping area of ​​the current static obstacle recognition area and the dynamic obstacle recognition area is greater than the preset area threshold, the three-dimensional position and occupied space data of the current static obstacle are not added; if the overlapping area of ​​the current static obstacle recognition area and the dynamic obstacle recognition area is less than or equal to the preset area threshold, the three-dimensional position and occupied space data of the current static obstacle are added to the initial three-dimensional cost map.

[0015] In a fourth exemplary embodiment, the first type of obstacle is a static obstacle, and the three-dimensional spatial data of the first type of obstacle is obtained by performing target recognition processing on the image data and the point cloud data of the driving area to be detected, including:

[0016] Extract two-dimensional image features from the image data, and identify the ground area contained in the two-dimensional image features; extract ground point cloud data corresponding to the ground area from the point cloud data according to the ground area and the calibration parameters between the laser radar and the camera, and obtain the ground plane according to the plane fitting result of the ground point cloud data; take the point cloud data in the point cloud data that is within a preset elevation threshold range as the projected point cloud data projected onto the ground with the ground plane as a reference; remove the ground point cloud data in the projected point cloud data to obtain non-ground point cloud data; obtain multiple obstacle areas according to the clustering result of the non-ground point cloud data, and obtain the three-dimensional position and occupied space of the corresponding obstacle area by fitting the point cloud data of each obstacle area, so as to serve as the three-dimensional spatial data of the static obstacle.

[0017] In a fifth exemplary embodiment, the first type of obstacle is a dynamic obstacle, and target recognition processing is performed on the image data and point cloud data of the driving area to be detected to obtain three-dimensional spatial data of the first type of obstacle, including:

[0018] An encoder-decoder network model is pre-trained; wherein the encoder is used to fuse input features, and the decoder is used to perform target recognition processing; the image data and point cloud data of the driving area to be detected are respectively converted to the target perspective to obtain point cloud features and image features; the point cloud features and image features are input to the encoder for fusion to obtain point graph fusion features; the point graph fusion features are input to the decoder to obtain the three-dimensional bounding box and posture angle of the dynamic obstacle as the three-dimensional spatial data of the dynamic obstacle.

[0019] In a sixth exemplary implementation, the point graph fusion features of the image data and the point cloud data of the driving area to be detected under the target perspective are updated according to the new grid elevation information corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement. The three-dimensional shape information of the second type of obstacle is determined based on the updated spatial voxel weight, including:

[0020] By performing target detection processing on the point map fusion features, the initial three-dimensional shape information of the obstacles in the driving area to be detected is obtained; according to the position information of the bird's-eye view grid area of ​​the obstacle identified in the current frame at the current moment, the grid contained in the obstacle area of ​​the current frame is determined; wherein the bird's-eye view grid area is the rasterized driving area to be detected; according to the new grid elevation information corresponding to the point cloud data in the obstacle area of ​​each frame continuously obtained during the vehicle's forward movement, the spatial voxel weight at the current moment is updated, and the elevation information and geometric shape data of the grid in the obstacle area are updated based on the updated spatial voxel weight, so as to continuously update the initial three-dimensional shape information.

[0021] In a seventh exemplary implementation, the step of performing target detection processing on the point image fusion features to obtain initial three-dimensional shape information of obstacles in the driving area to be detected includes:

[0022] A three-dimensional target detection network model is pre-built and trained; the three-dimensional target detection network model includes a detection task head, a classification task head and an instance segmentation task head, the detection task head is used to identify the second type of obstacles and output the second type of obstacle position information; the classification task head is used to identify the second type of obstacle category information; the instance segmentation task head is used to segment the second type of obstacles on the target plane to obtain instance segmentation results; the point map fusion feature is input into the three-dimensional target detection network model to obtain the position information, category information and instance segmentation results of the second type of obstacles; based on the instance segmentation results and the point cloud data in the obstacle area, the geometric morphology description data of the second type of obstacles is determined through the three-dimensional surface fitting results; the position information, category information and geometric morphology description data of the second type of obstacles are used as the initial three-dimensional shape information of the second type of obstacles.

[0023] In an eighth exemplary implementation, determining the grid included in the obstacle area of ​​the current frame according to the position information of the grid area of ​​the bird's-eye view of the obstacle identified in the current frame at the current moment includes:

[0024] According to the second-category obstacle recognition result corresponding to the current frame, the grid in the corresponding second-category obstacle area is determined; if the current point position is the position information of the second-category obstacle recognized in the current frame in the bird's-eye view grid area at the current moment, a first preset value is set for the current point as a mask value; if the current point position is not the position information of the second-category obstacle recognized in the current frame in the bird's-eye view grid area at the current moment, a second preset value is set for the current point as a mask value; according to the mask value of each point in the bird's-eye view grid area, obstacle area mask information is generated to identify the grids included in the obstacle area of ​​the current frame.

[0025] In a ninth exemplary implementation, the spatial voxel weight at the current moment is updated according to the new grid elevation information corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement, and the three-dimensional shape information of the second type of obstacle is determined based on the updated spatial voxel weight, including:

[0026] For each grid in the rasterized driving area to be detected, a target grid belonging to the second type of obstacle area identified in the current frame is obtained; if the current target grid does not belong to the radar point cloud projection area, the weight of the spatial voxel of the current target grid is determined based on the spatial voxel coordinate position of each target grid in the second type of obstacle area; if the current target grid belongs to the radar point projection area, the weight of the spatial voxel of the current target grid is determined based on the spatial voxel coordinate position of each target grid in the second type of obstacle area and the elevation information of the radar point projection area.

[0027] In a tenth exemplary implementation, determining the three-dimensional shape information of the second type of obstacle based on the updated spatial voxel weight includes:

[0028] Pre-train an elevation classification network, the elevation classification network is used to convert the depth estimation problem into a classification problem, and the elevation classification network determines the loss function based on the obstacle type, elevation truth encoding information, obstacle area mask information, point cloud features and image features of the second type of obstacle area grid, and spatial voxel weights of the second type of obstacle area grid; the elevation information of the grid in the second type of obstacle area is obtained through the elevation classification network.

[0029] The present invention also provides an electronic device, comprising a processor, wherein the processor is used to implement the steps of the cost map generation method as described in any of the preceding items when executing a computer program stored in a memory.

[0030] Finally, the present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the cost map generation method as described in any of the preceding items are implemented.

[0031] The present invention also provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the cost map generation method as described in any of the preceding items.

[0032] Finally, the present invention also provides an unmanned vehicle, including an automatic driving perception system, wherein the automatic driving perception system is used to implement the steps of the cost map generation method as described in any of the preceding items when executing a computer program stored in a memory.

[0033] The technical solution provided by the present invention has the advantage that the gridded area in front of the vehicle is used as the horizontal axis, and the spatial voxels set in the vertical direction of each grid in the horizontal plane are used as the vertical axis, so that a three-dimensional cost map can be constructed. By processing the multi-source sensor data accordingly, not only can the three-dimensional spatial data of large obstacles be obtained, such as the spatial position information of common dynamic obstacles and large-scale static obstacles, but also the three-dimensional position and morphological information of obstacles with small elevations can be accurately identified, such as low raised obstacles and potholes. The three-dimensional spatial data of multiple types of obstacles are filled and merged into the three-dimensional cost map, so that the three-dimensional data of various obstacles in front of the vehicle can be obtained, and then the obstacles can be accurately avoided, and the obstacle avoidance ability is effectively improved, which is conducive to the intelligent driving vehicle to complete the driving task efficiently and safely.

[0034] In addition, the present invention also provides corresponding electronic devices, non-volatile storage media, computer program products and unmanned vehicles for implementing the cost map generation method, further making the method more practical, and the electronic devices, non-volatile storage media, computer program products and unmanned vehicles have corresponding advantages.

[0035] The technical features mentioned above, the technical features to be mentioned below, and the technical features shown separately in the drawings can be arbitrarily combined with each other, as long as the combined technical features are not contradictory. All feasible feature combinations are technical contents clearly recorded in this article. Any of the multiple sub-features contained in the same sentence can be applied independently, and does not have to be applied together with other sub-features. It should be understood that the above general description and the detailed description below are only exemplary and cannot limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 is a schematic diagram of a cost map in the related art;

[0038] Figure 2 A schematic diagram of a flow chart of a cost map generation method provided by the present invention;

[0039] Figure 3 A schematic diagram of a rasterized driving area to be detected provided by the present invention;

[0040] Figure 4A schematic diagram of the display content of the three-dimensional cost map provided by the present invention;

[0041] Figure 5 A schematic diagram of point cloud data projected onto the ground provided by the present invention;

[0042] Figure 6 A schematic diagram of spatial voxels on a longitudinal section of a road surface provided by the present invention;

[0043] Figure 7 A side view schematic diagram of elevation estimation provided by the present invention;

[0044] Figure 8 A schematic diagram of geometric description data of obstacles provided by the present invention;

[0045] Fig. 9 A schematic diagram of a mask of an obstacle provided by the present invention;

[0046] Fig.10 A schematic diagram of the spatial voxel weights of obstacles provided by the present invention;

[0047] Fig.11 A schematic diagram of a flow chart of another cost map generation method provided by the present invention;

[0048] Fig.12 A schematic diagram of coordinate system definition in an exemplary example is provided for the present invention;

[0049] Fig.13 A structural diagram of a specific implementation of the cost map generation device provided by the present invention;

[0050] Fig.14 A structural diagram of a specific implementation of the electronic device provided by the present invention.

[0051] Fig.15 A schematic diagram of obstacle recognition for an unmanned vehicle provided by the present invention. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, the terms "first", "second", "third", "fourth", etc. in the specification and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations of the two are intended to cover non-exclusive inclusions. The term "exemplary" means "used as an example, embodiment or illustrative". Any embodiment described here as "exemplary" is not necessarily interpreted as being superior or better than other embodiments.

[0053] The cost map is used to store external sensor information obtained by sensors. This information is processed to generate a two-dimensional grid map. In the map, obstacle information will be converted to the corresponding grid and the obstacles will be expanded using corresponding rules.

[0054] In the related technologies, the cost map is two-dimensional plane information under the omniscient perspective, and does not take into account the existence of low obstacles / potholes and other fault information in front of the vehicle. It is impossible to accurately avoid such obstacles, and the obstacle avoidance ability is poor, resulting in the inability of intelligent driving vehicles to complete driving tasks efficiently and safely. At present, some related technologies are also unable to accurately identify such obstacles with small elevations. For example, a related technology realizes the detection of potholes on the road surface of unmanned vehicles through multi-sensor fusion. This method uses sound waves to scan the road surface in front. When a pothole is detected, the camera is used to capture the road surface image, and then the pre-processed image is threshold-segmented to obtain a binary image. The number of white pixels in the binary image is counted to determine whether there is water in the pothole. The width of the pothole edge from the road edge on both sides is calculated through the image information captured by the camera, and the minimum torque for the unmanned vehicle to pass through the pothole is calculated. Finally, the relationship between the minimum torque and the maximum torque is used to determine whether the unmanned vehicle can pass safely. The image binarization of this method is greatly affected by the threshold, and it is difficult to adapt to various lighting conditions. It is also impossible to provide information such as the depth of the pothole, and it is also impossible to accurately calculate the three-dimensional morphology information of the pothole area and low obstacles. Another related technology uses an RGB-D (Red Green Blue Depth, a color image with depth information) depth camera to detect low raised obstacles and potholes on the road. This method uses the original point cloud of the RGB-D depth camera for preprocessing to obtain the corresponding plane point cloud, extracts the raised low obstacle point cloud data from the plane point cloud, and projects the extracted raised low obstacle point cloud onto the plane model parameter plane, downsamples the raised low obstacle point cloud on the model parameter plane, performs convex hull segmentation, transforms the convex hull to the / map (map) coordinate system and publishes it, merges the projected raised low obstacle point cloud into the ground plane point cloud to form a new plane point cloud, transforms the new plane point cloud to base_footprint (robot body coordinate system) into a ground plane two-dimensional grid map, performs pseudo pothole filtering on the ground plane two-dimensional grid map, and extracts potholes using an edge extraction method. Since the effective working distance of RGB-D cameras is very limited, usually within 10 meters, and they are basically ineffective under strong outdoor light, the accuracy of the point cloud generated by the RGB-D camera is poor due to the influence of the projection density of the generator. Not only is the usage scenario limited, but the effectiveness of the actual use process remains to be verified, and the calculation accuracy of the three-dimensional morphology information of potholes and low obstacles cannot be ensured. Another related technology is based on neural networks and binocular stereo vision to detect potholes on the road. This method uses a binocular camera installed in front of the vehicle to obtain the left and right views of the road scene; the left view is detected using an object detection neural network.When the existence of potholes is detected, the left and right views are stereo matched to obtain a depth map. The pothole position information predicted by the neural network is used to cut out the depth map of the pothole and the road surface around the pothole. Finally, the road plane around the pothole is fitted using a combination of RANSAC and genetic algorithms to calculate the distance from the pothole point cloud to the fitting plane, and the pothole depth is obtained by summing and averaging the distances. The minimum circumscribed rectangle of the projection point is determined by projecting the pothole point cloud onto the fitting road plane, and the area of ​​the minimum circumscribed rectangle is determined to obtain the area of ​​the pothole. However, due to the existence of void points in the parallax, this method has a large estimation error in the depth direction, and it is difficult to obtain high-precision pothole depth and convex height, and thus it is impossible to accurately identify pothole areas and low obstacles, and it is also impossible to accurately determine the three-dimensional morphology information of pothole areas and low obstacles. It can be seen that when generating cost maps, the relevant technology cannot accurately identify such obstacles with small elevations, and the intelligent driving vehicle has poor obstacle avoidance capabilities, which leads to the inability of intelligent driving vehicles to meet actual needs in terms of safety and comfort.

[0055] In view of this, the present invention comprehensively considers various types of obstacles, and based on the image data and point cloud data collected by multi-source sensor data, it can not only obtain the three-dimensional spatial data of large obstacles, such as the spatial position information of common dynamic obstacles and large-scale static obstacles, but also accurately identify the three-dimensional position and morphological information of obstacles with small elevations, such as low raised obstacles and potholes, and fill and merge the three-dimensional spatial data of multiple types of obstacles into the three-dimensional cost map, so as to obtain three-dimensional data for accurately identifying various types of obstacles in front of the vehicle, and effectively improve the vehicle's obstacle avoidance ability. After introducing the technical scheme of the present invention, various non-limiting embodiments of the present invention are described in detail below. In order to better illustrate the present invention, many specific details are given in the specific embodiments below. Those skilled in the art should understand that the present invention can also be implemented without these specific details. In other examples, the methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present invention.

[0056] First see Figure 2 , Figure 2 A schematic diagram of a cost map generation method provided in this embodiment, this embodiment may include the following contents:

[0057] S201: constructing a blank three-dimensional cost map based on rasterization processing, wherein each grid includes a spatial voxel in a horizontal direction perpendicular to the road surface in a driving area to be detected.

[0058] Among them, a blank three-dimensional cost map refers to a cost map without any content. The road surface to be detected is used as the horizontal plane, and the x-axis or horizontal axis is established. The x-axis is oriented to the right as the positive direction, and the y-axis or vertical axis is oriented upward as the positive direction. The intersection of the x-axis and the y-axis is called the origin of the plane rectangular coordinate system. The z-axis is constructed in the vertical direction where the spatial voxels are located, so as to establish a three-dimensional cost map in the three-dimensional space. For example, based on the self-vehicle coordinate system, the driving area to be detected is 100 in length × 20 in width, in meters. Combined with the spatial range corresponding to the rasterized driving area to be detected, the three-dimensional cost map is defined as: 100 (X-axis direction) × 20 (Y-axis direction) × 8 (Z-axis direction), in meters.

[0059] In this embodiment, the driving area to be detected is located in front of the vehicle, and is a space area extending forward from the position of the front of the vehicle. Its size can be flexibly selected according to actual conditions. The vehicle can be any vehicle with an autonomous driving function, and is not limited to autonomous driving vehicles or unmanned vehicles. The vehicle can also be called an ego vehicle (the current autonomous driving vehicle). In the field of computer vision, a network structure is used to predict elevation information to replace monocular depth estimation, such as BEVheight (a visual three-dimensional object detection framework with height information designed based on roadside scenes) and RoadBEV (road surface reconstruction in a bird's-eye view). However, the error in the predicted elevation of this method can reach about 2 centimeters, and this error makes it impossible to accurately judge low raised obstacles and potholes. The present invention divides the driving area to be detected into multiple grids under the target perspective, that is, rasterization processing, such as Figure 3 As shown, the driving area to be detected in front of the vehicle is divided into multiple grids. Since the laser radar installed on the vehicle will emit laser light during the vehicle's driving process, Figure 3The black area in represents the projection point of the laser radar point cloud on the road surface, and the white area represents no data. The grid size and the number of grids can be flexibly selected according to the actual situation, such as the amount of calculation, the obstacle detection accuracy and the configuration parameters of the laser radar installed on the vehicle, which does not affect the implementation of the present invention. Among them, the obstacle is the object that the present invention needs to detect in the driving area to be detected, including but not limited to obtaining the spatial position information of common dynamic obstacles such as pedestrians and vehicles, large-scale static obstacles such as walls, bushes, and trees, and having a small height in the direction perpendicular to the horizontal plane of the road surface, including but not limited to low raised obstacles, such as speed bumps, raised cracks, and potholes, such as puddles, shallow pits, and deep pits. On the basis of constructing a three-dimensional cost map based on three-dimensional space, this embodiment also takes into account that there may be a certain angle between the road surface and the xy plane of the vehicle coordinate system. If this situation is not taken into account, the error will be introduced into the elevation calculation. For areas with smaller elevations, that is, obstacles such as low raised obstacles and potholes, misjudgment will be caused. Therefore, the present invention also provides each grid with spatial voxels in a horizontal direction perpendicular to the road surface. The spatial voxels are multiple spatial cubes arranged in a horizontal direction perpendicular to the road surface. The spatial voxels can reflect the ups and downs of the ground, which are used in the subsequent accurate calculation of the elevation information of low raised obstacles and potholes, so as to be used in roadside perception, smart transportation, V2X (vehicle to everything, information exchange between vehicles and the outside world), smart security, collaborative perception and other fields for obstacle avoidance, environmental perception, etc.

[0060] S202: Obtain three-dimensional spatial data of the first type of obstacles by performing target recognition processing on the image data and point cloud data of the driving area to be detected.

[0061] 35: Among them, the point cloud data is the point cloud data collected by the vehicle laser radar, and the image data is the image collected by the vehicle camera during the vehicle driving process. In order to improve the accuracy of determining the three-dimensional shape information of various obstacles, before obtaining the point cloud data and image data, the calibration of each sensor of the vehicle can be completed through the internal mechanism of the vehicle. The sensor includes but is not limited to the inertial measurement unit, odometer, laser radar, forward camera, and rear camera, and the coordinate system transformation relationship between the sensors is obtained. For example, the calibration parameters before the vehicle leaves the factory can be directly used. In order to further improve the accuracy of generating three-dimensional shape information of road obstacles, the time alignment of point cloud data and image data can be ensured through the data time synchronization mechanism.

[0062] 36: After obtaining the point cloud data and image data, any target detection algorithm, such as R-CNN (Region-based Convolutional Neural Networks), Fast R-CNN (a fast region-based convolutional neural network), Faster R-CNN (a faster region-based convolutional neural network) and YOLOv3 (You Only Look Once version 3, network model name), can be used to perform target detection. The target refers to the first type of obstacles that need to be identified in this embodiment, which may include but are not limited to obtaining the spatial position information of common dynamic obstacles such as pedestrians and vehicles, and large-scale static obstacles such as walls, bushes, and trees. After the first type of obstacles are identified by the target detection algorithm, the position and geometric data of the obstacle, such as direction, length, width, and area, can be further determined, and used as the three-dimensional spatial data of the identified first type of obstacles.

[0063] S203: Based on the point-graph fusion features of the image data and point cloud data of the driving area to be detected under the target perspective, the spatial voxel weight at the current moment is updated according to the new grid elevation information corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement, and the three-dimensional morphology information of the second type of obstacle is determined based on the updated spatial voxel weight.

[0064] After the image data and point cloud data are acquired, the acquired point cloud data and image data can be converted to the target perspective. The target perspective can be determined according to the method used by the three-dimensional morphological information actually used. For example, the target perspective is a bird's-eye view. For the convenience of description, the features obtained after the conversion are defined as point cloud features and image features. In order to further improve the second type of obstacles, the elevation information of the second type of obstacles is less than the elevation information of the first type of obstacles, that is, the second type of obstacles are a type of obstacles with a relatively small elevation, including but not limited to low raised obstacles, such as speed bumps, raised cracks, and potholes, such as puddles, shallow pits, and deep pits. When the S201 step divides the driving area to be detected into a plurality of grids with spatial voxels, and each grid is provided with spatial voxels, the inclination angle of the driving area to be detected and the horizontal plane of the vehicle coordinate system can be determined by a plane fitting method. For example, the point cloud data of each grid is plane fitted to obtain the corresponding three-dimensional plane equation, and the road surface normal vector is obtained according to the three-dimensional plane equation. The angle between the road surface normal vector and the horizontal plane of the vehicle coordinate system is the inclination angle. According to the installation position of the laser radar and the structural parameters of the vehicle, the distance from the origin of the laser radar coordinate system to the ground can be obtained. Based on the principle of triangular geometry, the x-coordinate of the intersection of the vehicle coordinate system x-axis and the road surface can be obtained according to the distance and the inclination angle. , the elevation information of the corresponding grid can be obtained according to the difference between the coordinate and the horizontal coordinate of the grid and the inclination angle. For the classic LSS (Lattice Sampling and Segmentation, a technology used for three-dimensional point cloud processing and scene understanding), in order to estimate the depth information of each pixel, it is generally believed that for each pixel, the corresponding grid is set along the light ray, and each grid does not have a weight setting similar to a priority. In order to accurately calculate the three-dimensional shape information of the second type of obstacle, this embodiment can further set weights for the spatial voxels. As the vehicle continues to move forward, that is, it continues to approach the second type of obstacle, such as a low obstacle / pothole area, the on-board sensor will continuously collect point cloud data. For each frame of point cloud data, new elevation information of each grid can be obtained, and the weight is continuously updated according to the new elevation information. Through the iterative mechanism, more accurate elevation information is gradually obtained, so that the elevation information and geometric data are continuously updated by updating the weight, and more accurate three-dimensional shape data of the second type of obstacle is obtained. The target elevation information calculation method based on point cloud data in this embodiment can effectively improve the depth calculation accuracy compared to LSS. In addition, the depth or elevation estimation method based on LSS has high requirements on the accuracy of the labeled data and is difficult to label. However, this embodiment can further improve the depth prediction accuracy by using a regression method based on the initial true value with a certain accuracy, without the need for a large amount of labeled data.

[0065] S204: Fill in a blank three-dimensional cost map according to the three-dimensional spatial data and the three-dimensional shape information to obtain a three-dimensional cost map.

[0066] Among them, the three-dimensional cost map is used to store external sensor information obtained by the vehicle's sensors, such as cameras, lidars, odometers, and inertial measurement units, and external sensor information such as image data and point cloud data. Through this information, the three-dimensional data of two types of obstacles are obtained through the above steps, and they are filled into the blank three-dimensional grid map of S201, and finally a three-dimensional grid map containing three-dimensional spatial data of different types of obstacles is generated. The navigation status of the intelligent driving vehicle can be obtained by occupying the grid. Each basic unit of the three-dimensional cost map has 255 different values, but only 3 states are needed at the bottom layer, occupied, idle and unknown. Each state has a special value assigned to the three-dimensional cost map.

[0067] In the technical solution provided in this embodiment, the gridded area in front of the vehicle is used as the horizontal axis, and the spatial voxels set in the vertical direction of each grid in the horizontal plane are used as the vertical axis, so that a three-dimensional cost map can be constructed. By processing the multi-source sensor data accordingly, not only can the three-dimensional spatial data of large obstacles be obtained, such as the spatial position information of common dynamic obstacles and large-scale static obstacles, but also the three-dimensional position and morphological information of obstacles with small elevations can be accurately identified, such as low raised obstacles and potholes. The three-dimensional spatial data of multiple types of obstacles are filled and merged into the three-dimensional cost map, so that the three-dimensional data of various obstacles in front of the vehicle can be obtained, and then the obstacles can be accurately avoided, and the obstacle avoidance ability is effectively improved, which is conducive to the intelligent driving vehicle to complete the driving task efficiently and safely.

[0068] When the three-dimensional cost map generated by the above embodiment provides classification data for dynamic and static obstacles in the scene, and the three-dimensional information of the second type of low obstacles / pothole areas is added to the three-dimensional cost map, considering that if there are ups and downs on the road during driving, such as in the underground parking lot scene, if this type of terrain information is not detected, it will affect the safety of the vehicle. In order to further improve the richness of the three-dimensional cost map and enhance the obstacle avoidance capability, this embodiment also provides an embodiment in which the three-dimensional cost map includes ground terrain data, which may include the following content:

[0069] According to the ground area determined by the image data and the calibration parameters between the lidar and the camera, the ground point cloud data corresponding to the ground area is extracted from the point cloud data; according to the clustering result of the average normal vector of each grid in the ground area, the ground area is segmented to obtain multiple sub-ground areas; according to the point cloud data contained in each sub-ground area, the corresponding sub-ground area is plane fitted to determine the ground elevation information and ground tilt information of each sub-ground, so as to obtain the three-dimensional ground data as the ground of the three-dimensional cost map.

[0070] This embodiment is to identify the situation of uphill and downhill encountered by the vehicle during driving. It can first perform target recognition or semantic segmentation processing on the image features corresponding to the image data collected by the camera to determine the ground area in the image, and then obtain calibration parameter information. The calibration parameter information of this embodiment refers to the calibration parameters between the laser radar and the camera. The ground area identified in the image is associated with the point cloud data, so that the ground point cloud data corresponding to the ground area can be extracted from the point cloud data. After obtaining the ground point cloud data, based on the grid setting of S201, each grid is , i and j represent the row and column positions of the grid area, such as grid(2,2) means located at Figure 3 Calculate the ground point cloud inside the grid in the second row and second column of the grid area in The local average normal vector , the local average normal vector inside all grid cells Clustering is performed. The clustering method can be, for example, the mean shift method or K-Means (the name of the clustering algorithm). The ground area is segmented according to the clustering results to obtain k road surface areas. , such as different uphill and downhill sections, for each road area , count the point cloud data contained therein, fit the plane of the road section, and obtain the elevation information and ground tilt information of the road surface. The ground tilt information includes but is not limited to the road surface inclination angle and roll angle.

[0071] The above embodiment does not limit how to generate a three-dimensional cost map that ultimately contains rich obstacle data. The present invention also provides an exemplary three-dimensional cost map generation method, which may include the following contents:

[0072] According to the three-dimensional ground data as the three-dimensional ground layer of the three-dimensional cost map, the blank three-dimensional cost map is filled to obtain an initial three-dimensional cost map; the three-dimensional spatial data of the first type of obstacles and the three-dimensional shape information of the second type of obstacles are added to the initial three-dimensional cost map to generate a three-dimensional cost map. The first type of obstacles include dynamic obstacles and static obstacles, and static obstacles may contain dynamic obstacles, so they need to be identified. Exemplarily, the three-dimensional spatial data filling process of the first type of obstacles can be: adding the three-dimensional bounding box and posture information of the dynamic obstacles to the initial three-dimensional cost map; for each static obstacle, if the overlapping area of ​​the current static obstacle recognition area and the dynamic obstacle recognition area is greater than the preset area threshold, the three-dimensional position and occupied space data of the current static obstacle are not added; if the overlapping area of ​​the current static obstacle recognition area and the dynamic obstacle recognition area is less than or equal to the preset area threshold, the three-dimensional position and occupied space data of the current static obstacle are added to the initial three-dimensional cost map.

[0073] After S201 constructs an empty 3D cost map, the 3D ground data is added to the 3D ground base layer. On the 3D ground data base layer, the 3D shape information of the second type of obstacles, i.e., low obstacles / potholes, is added, and the 3D bounding box data of dynamic obstacles such as pedestrians and vehicles is further added. Considering that the large-scale static obstacle results may contain dynamic obstacles, such as large vehicles and buses, when adding large-scale static obstacles, if the dynamic obstacle recognition area and the large-scale static area overlap to a large extent, for example, the intersection over union (IoU) can be used to measure the overlap between the static obstacle recognition area and the dynamic obstacle recognition area, and the preset area threshold is a preset empirical value, for example, 0.5 can be selected. Accordingly, when the IoU of the static obstacle recognition area and the dynamic obstacle recognition area is greater than 0.5, the large-scale static obstacle is not added; when the IoU of the static obstacle recognition area and the dynamic obstacle recognition area is less than or equal to 0.5, the large-scale static obstacle is added. Furthermore, in addition to marking the three-dimensional spatial data, the three-dimensional cost map of the present invention can also mark the categories of various obstacles, such as whether they are large-scale static obstacles, dynamic obstacles, or low obstacles / potholes, thereby ultimately generating a three-dimensional semantic-level cost map that integrates the recognition results of large-scale static obstacles, dynamic obstacles, low obstacles / potholes, and three-dimensional ground data. Figure 4 shown.

[0074] From the above, it can be seen that the three-dimensional cost map of this embodiment integrates the recognition results of large-scale static obstacles, dynamic obstacles, low obstacles / pothole areas, and three-dimensional ground data, and the overlapping area judgment method can avoid the same obstacle from being marked multiple times, thereby improving the accuracy and effectiveness of the three-dimensional cost map, which is beneficial to improving the vehicle's obstacle avoidance ability.

[0075] The above embodiment does not limit how to obtain the three-dimensional spatial data of the first type of obstacles. This embodiment also provides an exemplary implementation process of obtaining the three-dimensional spatial data by performing target recognition processing on the image data and point cloud data of the driving area to be detected in combination with different obstacle types, which may include the following contents:

[0076] For the first type of obstacles, which are large-scale static obstacles, two-dimensional image features are extracted from the image data, and the ground area contained in the two-dimensional image features is identified; according to the ground area and the calibration parameters between the lidar and the camera, the ground point cloud data corresponding to the ground area is extracted from the point cloud data, and the ground plane is obtained according to the plane fitting result of the ground point cloud data; with the ground plane as a reference, the point cloud data within the preset elevation threshold range in the point cloud data is used as the projected point cloud data projected to the ground; the ground point cloud data in the projected point cloud data is removed to obtain non-ground point cloud data; according to the clustering result of the non-ground point cloud data, multiple obstacle areas are obtained, and the three-dimensional position and occupied space of the corresponding obstacle area are obtained by fitting the point cloud data of each obstacle area, which is used as the three-dimensional spatial data of the static obstacle.

[0077] In the autonomous driving scene, large-scale static obstacles usually include bushes, trees, walls, etc. Usually, the lidar point cloud data will contain a large amount of ground point cloud data, which can easily affect the recognition accuracy of large-scale static obstacles. In order to accurately identify static obstacles, the ground point cloud data can be removed first. For example, any backbone network is used to extract two-dimensional image features from image data, and any semantic segmentation network, such as U-net (network name), is used to identify the ground area in the image features. Then, the calibration parameters of the lidar and the camera are used to associate the ground area identified in the image features with the point cloud data to obtain the ground point cloud data. Furthermore, in order to reduce the amount of data to be processed, point cloud data projected onto the ground can be further obtained. Any fitting method, such as the least squares method, can be used to fit the ground point cloud data. Fitting the ground plane , for the original point cloud data, As a benchmark, the point cloud data within the preset elevation threshold range is used as the point cloud data projected onto the ground, such as Figure 5 As shown, the preset elevation threshold range is For example, 5 cm can be used to project points within the elevation threshold range onto the ground as point cloud data. . Then from Eliminate ground point cloud , get the non-ground point cloud Any clustering method can be used for non-ground point clouds. Clustering is performed, and multiple regions can be obtained based on the clustering results. The three-dimensional position and occupied space of each region can be obtained through point cloud fitting. Of course, the large-scale static obstacles identified in this embodiment may include vehicles, pedestrians, etc. The semantic information of the incorrectly identified obstacles can be modified based on the dynamic obstacle recognition results combined with the overlapping area between the two, that is, the static obstacles are corrected to dynamic obstacles at the obstacle type semantic information mark of the three-dimensional cost map.

[0078] For the scene where the first type of obstacle is a dynamic obstacle, the corresponding three-dimensional spatial data acquisition process may include: pre-training the encoder-decoder network model; converting the image data and point cloud data of the driving area to be detected to the target perspective respectively to obtain point cloud features and image features; inputting the point cloud features and image features into the encoder for fusion to obtain point graph fusion features; inputting the point graph fusion features into the decoder to obtain the three-dimensional bounding box and posture angle of the dynamic obstacle as the three-dimensional spatial data of the dynamic obstacle.

[0079] In this embodiment, in the autonomous driving scene, dynamic obstacles may include pedestrians and vehicles moving or parked. Any decoder and encoder network structure can be used to construct the encoder-decoder network model of this embodiment. The encoder is used to fuse input features, and the decoder is used to perform target recognition processing. After obtaining the point cloud features and image features from the target perspective, the encoder fuses the two, and the decoder obtains the three-dimensional bounding box or three-dimensional encompassing box and attitude angles of the dynamic obstacle, such as pitch angle, yaw angle, etc.

[0080] The above embodiment does not limit how to perform rasterization processing on the to-be-detected driving area. Based on the above embodiment, the present invention further provides an exemplary rasterization processing implementation method of the to-be-detected driving area, which may include the following contents:

[0081] As an exemplary rasterization method, the number of laser radar lines installed on the autonomous driving vehicle can be obtained; the number of grids and the grid size are determined according to the number of laser radar lines, and the spatial size of the driving area to be detected is determined according to the grid data and the grid size; according to the number of grids and the grid size, the driving area to be detected within the corresponding spatial size in front of the driving direction is rasterized.

[0082] As another exemplary rasterization method, the resolution of the laser radar installed on the autonomous driving vehicle can be obtained; the number of grids and the grid size are determined according to the resolution of the laser radar, and the spatial size of the driving area to be detected is determined according to the grid data and the grid size; according to the number of grids and the grid size, the driving area to be detected within the corresponding spatial size in front of the driving direction is rasterized.

[0083] In this embodiment, the number of grids in the bird's-eye view grid area and the size of the space corresponding to each grid on the road to be detected can be flexibly adjusted according to the number of laser radar lines or resolution. For example, the vehicle is equipped with a 32-line mechanical laser radar. Considering the accuracy and calculation amount, the number of grids in the bird's-eye view grid area is 2000×2000, and the space size corresponding to each grid is 0.01 meter×0.01 meter. The length and width of the space area corresponding to the bird's-eye view grid area are 20 meters and 20 meters respectively. For more than 20 meters, it is a far-field detection area. If the vehicle is equipped with a 128-line mechanical laser radar, the size of the space corresponding to the bird's-eye view grid area can be increased to 40 meters×30 meters. In order to improve the detection and recognition accuracy, the size of the space corresponding to each grid can be set to 0.01 meter×0.01 meter.

[0084] In order to further improve the recognition accuracy of road obstacles, based on the above embodiments, the present invention further provides a method for determining parameters of spatial voxels of a grid, which may include the following contents:

[0085] Obtain the maximum elevation of obstacles in the driving area to be detected; divide the maximum elevation of obstacles according to a preset accuracy threshold to determine the total number of spatial cubes contained in the spatial voxels of each grid; determine the real physical space size corresponding to the spatial cube of the spatial voxels of each grid based on the maximum elevation of obstacles, the total number of spatial cubes and the preset plane size of spatial cubes.

[0086] In this embodiment, if Figure 6 As shown, Figure 6 The left side is a schematic diagram of space voxels, which are 2t space cubes, and the right side is a longitudinal section of the road surface. is the elevation difference between the road surface where the spatial voxel is located and the coordinate system of the autonomous driving vehicle, that is, the horizontal plane of the vehicle coordinate system. When the road surface is horizontal or the pitch angle between the vehicle coordinate system and the road surface is 0, 0. However, since the road surface itself may be inclined or when the vehicle passes through a low raised obstacle or a pothole area, there is a certain pitch angle between the vehicle coordinate system and the ground. In order to more accurately obtain the elevation of the low raised obstacle or pothole area, Values ​​such as Figure 6 As shown in the figure on the right, Take the elevation difference between the road surface where the spatial voxel is located and the horizontal plane of the vehicle coordinate system.

[0087] The number of space cubes contained in the space voxel is determined by the maximum elevation of the obstacle to be identified, such as a low raised obstacle / pothole area. For example, the total number of space cubes can be determined by the multiple of the number of segments of the maximum elevation of the obstacle, and the real physical space height corresponding to the space cube can be determined by the ratio of the maximum elevation of the obstacle to the total number of space cubes. The real physical space size corresponding to the space cube of the space voxel of each grid is determined according to the real physical space height and the grid size. For example, the space voxel may include A space cube, The maximum elevation of the low raised obstacle / pothole area to be identified is set. For example, the maximum elevation of the low raised obstacle / pothole area to be identified is 0.15 meters. In order to accurately identify the elevation information of each obstacle such as a low raised obstacle, the maximum elevation, that is, 15 centimeters, can be divided according to the preset accuracy threshold. The preset accuracy threshold can be selected as 1 cm, so 15 centimeters is divided into 1 cm, and the corresponding , the total number of space cubes contained in the space voxel is 2×15=30. For example, in order to facilitate data processing, the length and width of the space cube correspond to the same space size as the length and width of the unit grid, such as 0.01 meters × 0.01 meters. Correspondingly, the height of the space cube corresponds to the real space size Meters, so the real space size of each stereo frame is 0.01×0.01×0.005, in meters.

[0088] As can be seen from the above, this embodiment sets the parameters of the grid area according to the laser radar parameters of the vehicle, which is conducive to improving the recognition accuracy of the second type of road obstacles. The spatial voxel parameters are set by the maximum elevation of the second type of obstacles. Since the spatial voxels are used to calculate the elevation information of the second type of obstacles, the recognition accuracy of this type of obstacles can be further improved.

[0089] The above embodiment does not limit how to obtain grid elevation information. Based on the above embodiment, the present invention also provides an exemplary implementation method, which may include the following contents:

[0090] According to the inclination data of the horizontal plane of the driving area to be detected and the vehicle coordinate system and the position information of each grid, the elevation difference between the road surface where the spatial voxels of each grid are located and the horizontal plane of the vehicle coordinate system is determined.

[0091] Exemplarily, the driving area to be detected is divided into multiple types of road areas according to the inclination of the road surface corresponding to each grid, and the inclination of the same type of road area meets the preset same similar inclination conditions; based on the inclination data of each road area and the horizontal plane of the vehicle coordinate system and the position information of each grid, the elevation difference between the road surface where the spatial voxels of each grid in the bird's-eye view grid area are located and the horizontal plane of the vehicle coordinate system is determined. Among them, the point cloud data in each grid is plane fitting processed to obtain the corresponding three-dimensional space plane equation; the angle between the normal vector of the three-dimensional space plane equation of each grid and the normal vector of the horizontal plane of the vehicle coordinate system is used as the plane angle between the road area of ​​the corresponding grid and the horizontal plane of the vehicle coordinate system. Figure 7 As shown in the figure, the coordinate system vertical distance H between the origin of the laser radar coordinate system and the road surface is determined according to the laser radar installation position and the vehicle result parameters, and the coordinate system vertical distance H and the plane angle , determine the position information of the intersection between the horizontal coordinate axis of the vehicle coordinate system and the road surface corresponding to each grid . The road area division process may include: fitting the three-dimensional space plane equation of the corresponding grid area according to the three-dimensional position information of the point cloud data inside each grid; determining the average normal vector in each grid based on the three-dimensional space plane equation of each grid; and segmenting the to-be-detected driving area according to the spatial position correspondence between the bird's-eye view grid area and the to-be-detected driving area according to the clustering result of the average normal vector of each grid, so as to obtain multiple types of road areas. For each grid, determine the position information of the intersection of the horizontal axis of the vehicle coordinate system and the road surface corresponding to the current grid on the horizontal axis; based on the plane angle, calculate the elevation difference between the road surface where the spatial voxels of each grid in the bird's-eye view grid area are located and the horizontal plane of the vehicle coordinate system according to the position information of each grid on the horizontal axis and the position information of the corresponding intersection on the horizontal axis. As an efficient calculation method, the elevation calculation relationship can be stored locally in advance, and the elevation information of the spatial voxels corresponding to each grid (i, j) corresponding to the section of the road surface can be calculated by calling the elevation calculation relationship. , the elevation calculation formula can be expressed as:

[0092] ;

[0093] Where i and j represent the grid in the i-th row and j-th column in the bird's-eye view grid area, such as (2, 2) Figure 3 The grid in the second row and second column of the grid area.

[0094] As can be seen from the above, this embodiment divides the road surface according to the inclination angle, and determines the road surface inclination angle in a simple manner, thereby effectively improving the determination accuracy of the elevation information of the obstacle.

[0095] The above embodiment does not limit how to obtain the three-dimensional shape information of the second type of obstacles. Based on the above embodiment, the present invention further provides an exemplary implementation method, which may include the following contents:

[0096] By performing target detection processing on the point map fusion features, the initial three-dimensional shape information of the obstacles in the driving area to be detected is obtained; according to the position information of the obstacle identified in the current frame in the bird's-eye view grid area at the current moment, the grid contained in the obstacle area of ​​the current frame is determined; wherein the bird's-eye view grid area is the driving area to be detected after rasterization processing; according to the new grid elevation information corresponding to the point cloud data in the obstacle area of ​​each frame continuously obtained during the vehicle's forward movement, the spatial voxel weight at the current moment is updated, and the elevation information and geometric shape data of the grid in the obstacle area are updated based on the updated spatial voxel weight, so as to continuously update the initial three-dimensional shape information.

[0097] After obtaining the multimodal features under the target perspective, in order to further improve the recognition accuracy of obstacles, such as low obstacles and potholes, point cloud features and image features can be fused. Based on the fused features, any target detection algorithm, such as R-CNN (Region-based Convolutional Neural Networks), Fast R-CNN (a fast region-based convolutional neural network), Faster R-CNN (a faster region-based convolutional neural network) and YOLOv3 (You Only Look Once version 3, network model name), is used to perform target detection on the fused features. The target refers to the second type of obstacles that need to be identified in this embodiment. The obstacles in the fused features identified by the target detection algorithm can further determine the elevation information, position, and geometric data of the obstacles, such as direction, length, width, and area, and use them as the three-dimensional shape information of the identified obstacles. Since the three-dimensional shape information determined in this step is not accurate, the three-dimensional shape information will be continuously updated through the data collected by the vehicle continuously approaching the obstacle, so this step defines it as the initial three-dimensional shape information.

[0098] As a high-precision method for determining three-dimensional shape information, this embodiment uses the excellent learning ability of the three-dimensional target detection network model to calculate the three-dimensional shape data of the obstacle, which may include the following contents:

[0099] A three-dimensional target detection network model is pre-built and trained; the three-dimensional target detection network model includes a detection task head, a classification task head and an instance segmentation task head, the detection task head is used to identify the second type of obstacles and output the second type of obstacle position information; the classification task head is used to identify the second type of obstacle category information; the instance segmentation task head is used to segment the second type of obstacles on the target plane to obtain instance segmentation results; the point map fusion features are input into the three-dimensional target detection network model to obtain the position information, category information and instance segmentation results of the second type of obstacles; based on the instance segmentation results and the point cloud data in the obstacle area, the geometric morphology description data of the second type of obstacles is determined through the three-dimensional surface fitting results; the position information, category information and geometric morphology description data of the second type of obstacles are used as the initial three-dimensional shape information of the second type of obstacles.

[0100] In this embodiment, the three-dimensional target detection network model can be trained based on any deep learning three-dimensional target detection algorithm. It may include a detection task head, a classification task head and an instance segmentation task head. The detection task head is used to identify obstacles and output obstacle location information; the classification task head is used to identify obstacle category information; the instance segmentation task head is used to segment obstacles on the target plane to obtain instance segmentation results. The point cloud features and image features under the target perspective are fused by convolution using a decoder-encoder or decoder mechanism or direct convolution. For fused features, features are first extracted through the backbone network and the neck module, and then the detection task head is used to obtain the location information of low obstacles / pothole areas; the classification task head is used to obtain obstacle category information, such as low obstacles, pits, cracks, road bumps, etc. Finally, the instance segmentation task head is used to obtain the segmentation results of obstacles such as low obstacles and pothole areas on the horizontal plane. Based on the segmentation results and its internal point cloud data, the geometric morphological description data of the obstacles can be obtained through a three-dimensional surface fitting method, such as Figure 8 As shown, the geometric description data can be long ,Width , the angle between the lane line and the , Figure 8 is a top view, is the x-axis of the vehicle coordinate system, is the y-axis of the vehicle coordinate system.

[0101] After the driving area to be detected is rasterized, if it is calculated point by point, the amount of calculation is huge. For example, if the number of grids is 2000×2000, and the voxel is set to 60 elevation estimates, the amount of calculation will be very large. Usually, obstacles will only occupy part of the grids. For non-low obstacles and potholes, it is not necessary to calculate their elevation information. In order to reduce the amount of calculation, this embodiment can also first determine the grids contained in the obstacles, and only calculate the elevation information of the grids occupied by obstacles. Among them, the grids occupied by obstacles include low obstacles / potholes identified in the previous frame, low obstacles / potholes identified in the current frame, and suspected low obstacles / potholes identified in the far field that enter the near field area. In order to further effectively reduce the amount of calculation, this embodiment also generates mask information based on the obstacle occupation grid, and determines the grid in the corresponding second-type obstacle area according to the second-type obstacle recognition result corresponding to the current frame; if the current point position is the position information of the second-type obstacle recognized in the current frame at the current moment in the bird's-eye view grid area, a first preset value is set for the current point as the mask value; if the current point position is not the position information of the second-type obstacle recognized in the current frame at the current moment in the bird's-eye view grid area, a second preset value is set for the current point as the mask value; according to the mask value of each point in the bird's-eye view grid area, the obstacle area mask information is generated to identify the grid contained in the obstacle area of ​​the current frame.

[0102] In this embodiment, the mask information is automatically generated based on the radar projection point and the position of the low obstacle / pothole area identified in the current frame in the grid area at the current moment, and does not require manual setting. Fig. 9 As shown, Fig. 9 Different lines in the grid are used to represent different types of second-class obstacles, such as speed bumps, deep pits, shallow pits, etc. The first-class area in the grid in the figure is the projection point of the LiDAR point cloud on the road surface, that is, the intersection of the LiDAR line beam and the ground. If there are multiple LiDAR point cloud projection points in the same grid, the z coordinate is the average value; the second-class area in the figure is the area where the obstacle is identified in the current frame.

[0103] For example, as a more efficient method, a pre-stored mask relationship can be called to generate corresponding mask information. , the mask relation can be expressed as:

[0104] ;

[0105] Among them, i and j represent the row and column positions of the grid, and the set D includes: the position of the obstacle identified in the current frame in the grid area at the current moment.

[0106] In the field of computer vision, in order to solve the problems existing in monocular depth estimation, related technologies use a network structure to predict elevation information to replace monocular depth estimation, such as BEVheight (a visual three-dimensional object detection framework with height information designed based on roadside scenes) and RoadBEV (reconstruction of road surface in bird's-eye view). However, the error in predicted elevation of this method can reach about 2 cm, and this error makes it impossible to accurately judge low raised obstacles and potholes. The reason is that the accuracy of the annotation data, especially the accuracy of elevation information annotation, is difficult to meet. Accurate annotation of pixel-level elevation information is very time-consuming and the effect is difficult to control. Therefore, this embodiment provides a method for accurately obtaining elevation information guided by dense point clouds, which may include the following:

[0107] The spatial voxels of each grid in the rasterized driving area to be detected and the corresponding initial weight values ​​are obtained; the elevation information of the grid corresponding to the current frame point cloud data is obtained as the new elevation information, and the weight values ​​of the spatial voxels at the corresponding positions are updated using the new elevation information.

[0108] In this embodiment, if Fig. 9 As shown in the figure, there are multiple areas in the driving area to be detected, that is, grid area types, where the weight distribution of different grids (i, j) in a region is different due to their elevation. Different, elevation It is the elevation information of each grid (i, j) in the area. The height of the real road surface corresponding to each grid may be different, so the weight distribution of each grid will also be different. The grid area type includes the position of the low obstacle / pothole area identified in the current frame in the grid area at the current moment and whether there is radar projection information at the same time. In order to further improve the calculation accuracy of the elevation information, this embodiment determines different weights according to different grid area types. For example, for each grid in the rasterized driving area to be detected, the type of each grid area in the rasterized road surface area to be detected can be obtained first, and the target grid in the second type of obstacle area identified in the current frame can be obtained; if the current target grid does not belong to the radar point cloud projection area, the weight of the spatial voxel of the current target grid is determined based on the spatial voxel coordinate position of each target grid in the second type of obstacle area; if the current target grid belongs to the radar point projection area, the weight of the spatial voxel of the current target grid is determined based on the spatial voxel coordinate position of each target grid in the second type of obstacle area and the elevation information of the radar point projection area. For example, if Fig.10 As shown, the coordinate range of the spatial voxel is -30~30. For different grid areas, the weight of the spatial voxel is calculated as follows:

[0109] For the position of the low obstacle / pothole area identified in the current frame in the bird's-eye view grid area at the current moment, the corresponding spatial voxels are also divided into two categories. The first type of grid area has no radar point cloud projection on its corresponding bird's-eye view grid, and the second type of grid area has radar point cloud projection on its corresponding bird's-eye view grid. The spatial voxel also has radar point cloud projection in the bird's-eye view grid. Correspondingly, the elevation information of the radar point cloud projection is , the weight calculation of the first type of grid area can be realized by the following relationship (1), and the weight calculation of the second type of grid area can be realized by the following relationship (2):

[0110] ; (1)

[0111] . (2)

[0112] For example, as a simple weight setting method, a one-dimensional Gaussian function relationship can be used to generate a weight calculation relationship based on the preset Gaussian curve control parameter value, the center position coordinates determined by different grid area types, and the grid space voxel position information as variables, so as to determine the weight value of each grid space voxel by calling the weight calculation relationship. Can be 1.0, Can be 1.0, the weight of the spatial voxel can be correspondingly a one-dimensional Gaussian curve express, is the center position of the curve, g(x) represents the Gaussian curve, and x in this embodiment represents the coordinate value of the spatial voxel.

[0113] When the weight calculation method of different grid areas is determined, the elevation information of the grid in the obstacle area is obtained through the elevation classification network. Among them, the elevation classification network is a pre-trained elevation classification network, which is used to convert the depth estimation problem into a classification problem, and the elevation classification network determines the loss function based on the obstacle type, elevation true value encoding information, obstacle area mask information, point cloud features and image features of the obstacle area grid, and spatial voxel weights of the obstacle area grid.

[0114] In this embodiment, based on the LSS method, the depth estimation problem is converted into a classification problem, and the elevation information of each bird's-eye view grid corresponding to the obstacle inside the road area to be detected can be obtained through the elevation classification network. In order to further improve the accuracy of the elevation information, this embodiment also provides a corresponding loss function. As an efficient implementation method, the loss function can be pre-stored. The loss function is determined based on the obstacle type, elevation truth encoding information, obstacle area mask information, point cloud features and image features of the obstacle area grid, and spatial voxel weights of the obstacle area grid. The elevation truth encoding information is processed using one-hot encoding, which can be expressed in the following mathematical form:

[0115] ;

[0116] in, , is the number of types of obstacles identified, is a one-hot GT mask (elevation truth mask), which is a mask generated using pre-labeled elevation truth values. is the mask information of the obstacle, For the A bird's-eye view grid is used to obtain point cloud features and image features. The loss function is based on common knowledge in the field of deep learning. Since the base does not affect the model training process, it is usually ignored.

[0117] It can be seen from the above that this embodiment generates obstacle mask information by utilizing multimodal features and the current frame recognition results, and sets different weights, which can not only effectively reduce the amount of calculation, but also help improve the accuracy of the three-dimensional shape data of the obstacle.

[0118] In order to make the technical solution of the present invention more clear to those skilled in the art, the present invention also provides an exemplary implementation mode, as shown in FIG. Fig.11 As shown, it may include the following:

[0119] In this embodiment, for data processing, the present invention also defines the coordinate system involved accordingly, such as Fig.12 As shown in the figure, the vehicle includes a far-field camera, a near-field camera, a lidar, an odometer, and an inertial measurement unit. For the convenience of drawing, the far-field camera and the near-field camera are not distinguished, and the odometer and inertial measurement unit are not drawn. Fig.12 middle, is the camera coordinate system, Along the camera optical axis. is the standard camera coordinate system, which is Along Rotation, where Vertically downward. is the laser radar coordinate system, is the local world coordinate system, which is defined on the road surface, where Vertically upward, For horizontal forward, To the left horizontally, the vehicle coordinate system can usually adopt the lidar coordinate system, and the local world coordinate system is the coordinate system of the road surface on which the vehicle is traveling.

[0120] This embodiment can pre-build a three-dimensional cost map generation model, and the user inputs the following data into the three-dimensional cost map generation model: near-field dense point cloud or single-frame point cloud data after distortion elimination, forward camera image data, and sensor calibration parameters, wherein the sensor calibration parameters include calibration parameters between the camera and the laser radar. Among them, for mechanical laser radar point cloud data, within a frame of data acquisition cycle, each frame of point cloud data has motion distortion due to vehicle motion. The distortion calibration method can adopt any method, such as obtaining the kinematic data of the vehicle, such as assuming uniform linear motion, estimating the vehicle's movement distance and direction within 100 milliseconds, and based on this, combined with the timestamp of each point, calibrate the point cloud xyz coordinates. Near-field dense point cloud refers to splicing two adjacent frames of point cloud data, that is, splicing the point cloud data of the current frame to the point cloud data obtained by the previous frame. If the previous frame of data is not the first frame of point cloud data, it is also a spliced ​​point cloud feature, thereby solving the problem of point cloud data sparsity.

[0121] The three-dimensional cost map generation model converts the input point cloud data into point cloud bev features. An exemplary conversion process may be: using the feature extraction and Neck modules of the point cloud target detection algorithm, such as Pointpillars (algorithm name), Second (algorithm name), and Centerpoint (algorithm name), to obtain the features of the point cloud features at the target perspective, such as the bird's-eye view, which can also be called the point cloud Bev (bird's-eye) features. The process of converting image data into image bev features: first, any backbone network, such as resnet50 (network name), can be used to extract the two-dimensional image features of the image data; then, based on the calibration parameters between the camera and the laser radar, the LSS algorithm or the cross attention method is used to convert the two-dimensional image features into image features at the target perspective, such as the features at the bird's-eye view, which can also be defined as image bev features. The two-dimensional image features are used to identify the ground area in the image using the method described in the above embodiment, and the ground point cloud data and non-ground point cloud data are obtained through the ground area. Static obstacles are identified by clustering the non-ground point cloud data, and three-dimensional ground information is obtained by fitting the ground point cloud data. Then the point cloud bev features and image bev features are fused through the encoder to obtain bev fusion features, and then the three-dimensional data of dynamic obstacles is obtained through the decoder. Then, mask information is constructed according to the decoder output and the initial three-dimensional spatial information of the second type of obstacles is determined. The final three-dimensional shape data of the second type of obstacles is obtained through weight update and elevation calculation. Then, the three-dimensional ground data, the three-dimensional spatial data of static obstacles, the dynamic three-dimensional spatial data, the three-dimensional shape information of the second type of obstacles and the corresponding obstacle types are fused. The final three-dimensional cost map generation model output is: three-dimensional cost map.

[0122] From the above, it can be seen that this embodiment is based on the image data and point cloud data collected by multi-source sensors. On the one hand, it can obtain the spatial position information of common dynamic obstacles such as pedestrians and vehicles, and on the other hand, it can accurately identify the three-dimensional position and morphology of low obstacles / potholes. At the same time, it can also obtain large-scale static obstacles and ground terrain information. Finally, multiple types of information are merged to construct a three-dimensional semantic-level cost map, which effectively improves the vehicle's obstacle avoidance capability.

[0123] It should be noted that there is no strict order of execution between the steps in the present invention. As long as they comply with the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 2-Figure 11 This is just a schematic and does not mean that this is the only execution order.

[0124] The present invention also provides a corresponding device for the cost map generation method, which further makes the method more practical. Among them, the device can be described from the perspective of functional modules and hardware. The cost map generation device provided by the present invention is introduced below. The device is used to implement the cost map generation method provided by the present invention. In this embodiment, the cost map generation device may include or be divided into one or more program modules, and the one or more program modules are stored in a storage medium and executed by one or more processors to complete the cost map generation method disclosed in Example 1. The program module referred to in this embodiment refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for describing the execution process of the cost map generation device in the storage medium than the program itself. The following description will specifically introduce the functions of each program module of this embodiment. The cost map generation device described below and the cost map generation method described above can be referenced to each other.

[0125] From the perspective of functional modules, see Fig.13 , Fig.13 The structure diagram of a cost map generation device provided in this embodiment under a specific implementation mode, the device may include:

[0126] The blank three-dimensional map construction module 131 is used to construct a blank three-dimensional cost map based on rasterization processing, wherein each grid includes a spatial voxel in a horizontal direction perpendicular to the road surface in the to-be-detected driving area.

[0127] The first type obstacle data acquisition module 132 is used to obtain the three-dimensional space data of the first type obstacles by performing target recognition processing on the image data and point cloud data of the driving area to be detected.

[0128] The second type obstacle data acquisition module 133 is used to update the spatial voxel weight at the current moment based on the point-graph fusion features of the image data and point cloud data of the driving area to be detected under the target perspective, according to the new grid elevation information corresponding to each frame of point cloud data continuously acquired during the vehicle's forward movement, and determine the three-dimensional morphology information of the second type of obstacle based on the updated spatial voxel weight.

[0129] The map generation module 134 is used to fill in a blank three-dimensional cost map according to the three-dimensional spatial data and the three-dimensional shape information to obtain a three-dimensional cost map; wherein the elevation information of the second type of obstacles is smaller than the elevation information of the first type of obstacles.

[0130] Exemplarily, in some implementations of the present embodiment, the above-mentioned device may also include a ground data acquisition module, which is used to: extract ground point cloud data corresponding to the ground area from the point cloud data according to the ground area determined by the image data and the calibration parameters between the laser radar and the camera; segment the ground area according to the clustering results of the average normal vectors of each grid in the ground area to obtain multiple sub-ground areas; perform plane fitting processing on the corresponding sub-ground area according to the point cloud data contained in each sub-ground area, and determine the ground elevation information and ground inclination information of each sub-ground to obtain three-dimensional ground data as the ground of the three-dimensional cost map.

[0131] As an exemplary implementation of the above embodiment, the above map generation module 134 can be further used to: fill in the blank three-dimensional cost map according to the three-dimensional ground data as the three-dimensional ground layer of the three-dimensional cost map to obtain an initial three-dimensional cost map; add three-dimensional spatial data and three-dimensional morphology information to the initial three-dimensional cost map to generate a three-dimensional cost map.

[0132] Exemplarily, in some other implementations of this embodiment, the map generation module 134 may be further used to: add the three-dimensional bounding box and posture information of the dynamic obstacle to the initial three-dimensional cost map; for each static obstacle, if the overlapping area of ​​the current static obstacle recognition area and the dynamic obstacle recognition area is greater than the preset area threshold, the three-dimensional position and occupied space data of the current static obstacle are not added; if the overlapping area of ​​the current static obstacle recognition area and the dynamic obstacle recognition area is less than or equal to the preset area threshold, the three-dimensional position and occupied space data of the current static obstacle are added to the initial three-dimensional cost map.

[0133] Illustratively, in some other implementations of the present embodiment, the first-category obstacle data acquisition module 132 may be further used to: extract two-dimensional image features from image data, and identify the ground area contained in the two-dimensional image features; extract ground point cloud data corresponding to the ground area from the point cloud data according to the ground area and the calibration parameters between the laser radar and the camera, and obtain the ground plane according to the plane fitting result of the ground point cloud data; take the point cloud data in the point cloud data that is within a preset elevation threshold range as the projected point cloud data projected onto the ground, with the ground plane as a reference; remove the ground point cloud data in the projected point cloud data to obtain non-ground point cloud data; obtain multiple obstacle areas according to the clustering result of the non-ground point cloud data, and obtain the three-dimensional position and occupied space of the corresponding obstacle area by fitting the point cloud data of each obstacle area as the three-dimensional spatial data of the static obstacle.

[0134] Exemplarily, in some other implementations of the present embodiment, the above-mentioned first-category obstacle data acquisition module 132 can be further used for: pre-training an encoder-decoder network model; wherein the encoder is used to fuse input features, and the decoder is used to perform target recognition processing; respectively converting the image data and point cloud data of the driving area to be detected to the target perspective to obtain point cloud features and image features; inputting the point cloud features and image features into the encoder for fusion to obtain point graph fusion features; inputting the point graph fusion features into the decoder to obtain the three-dimensional bounding box and posture angle of the dynamic obstacle as the three-dimensional spatial data of the dynamic obstacle.

[0135] Exemplarily, in some other implementations of the present embodiment, the second-category obstacle data acquisition module 133 may be further used to: obtain initial three-dimensional shape information of obstacles in the driving area to be detected by performing target detection processing on the point map fusion features; determine the grid contained in the obstacle area of ​​the current frame according to the position information of the bird's-eye view grid area of ​​the obstacle identified in the current frame at the current moment; wherein the bird's-eye view grid area is the rasterized driving area to be detected; update the spatial voxel weight at the current moment according to the new grid elevation information corresponding to the point cloud data in the obstacle area of ​​each frame continuously acquired during the vehicle's forward movement, and update the elevation information and geometric morphology data of the grid in the obstacle area based on the updated spatial voxel weight, so as to continuously update the initial three-dimensional shape information.

[0136] As an exemplary implementation of the above embodiment, the above-mentioned second-category obstacle data acquisition module 133 can also be further used to: pre-build and train a three-dimensional target detection network model; the three-dimensional target detection network model includes a detection task head, a classification task head and an instance segmentation task head, the detection task head is used to identify the second-category obstacles and output the second-category obstacle position information; the classification task head is used to identify the second-category obstacle category information; the instance segmentation task head is used to segment the second-category obstacles on the target plane to obtain instance segmentation results; the point map fusion features are input into the three-dimensional target detection network model to obtain the position information, category information and instance segmentation results of the second-category obstacles; based on the instance segmentation results and the point cloud data in the obstacle area, the geometric morphology description data of the second-category obstacles is determined through the three-dimensional surface fitting results; the position information, category information and geometric morphology description data of the second-category obstacles are used as the initial three-dimensional morphology information of the second-category obstacles.

[0137] As another exemplary implementation of the above embodiment, the above second-category obstacle data acquisition module 133 can be further used to: determine the grid in the corresponding second-category obstacle area according to the second-category obstacle recognition result corresponding to the current frame; if the current point position is the position information of the second-category obstacle recognized in the current frame in the bird's-eye view grid area at the current moment, set a first preset value for the current point as a mask value; if the current point position is not the position information of the second-category obstacle recognized in the current frame in the bird's-eye view grid area at the current moment, set a second preset value for the current point as a mask value; generate obstacle area mask information according to the mask values ​​of each point in the bird's-eye view grid area to identify the grids contained in the obstacle area of ​​the current frame.

[0138] Exemplarily, in some other implementations of the present embodiment, the second-category obstacle data acquisition module 133 may be further used to: for each grid in the rasterized driving area to be detected, obtain the target grid in the second-category obstacle area identified in the current frame; if the current target grid does not belong to the radar point cloud projection area, then determine the weight of the spatial voxel of the current target grid based on the spatial voxel coordinate position of each target grid in the second-category obstacle area; if the current target grid belongs to the radar point projection area, then determine the weight of the spatial voxel of the current target grid based on the spatial voxel coordinate position of each target grid in the second-category obstacle area and the elevation information of the radar point projection area.

[0139] Exemplarily, in some other implementations of the present embodiment, the above-mentioned second-category obstacle data acquisition module 133 can be further used for: pre-training an elevation classification network, the elevation classification network is used to convert the depth estimation problem into a classification problem, and the elevation classification network is based on the obstacle type, elevation true value coding information, obstacle area mask information, point cloud features and image features of the second-category obstacle area grid, and the spatial voxel weights of the second-category obstacle area grid to determine the loss function; the elevation information of the grid in the second-category obstacle area is obtained through the elevation classification network.

[0140] From the above, it can be seen that the cost map generated by this embodiment contains richer information and can effectively improve the obstacle avoidance capability of the intelligent driving vehicle.

[0141] The cost map generation device mentioned above is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, which is described from the perspective of hardware. Fig.14 The following is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention in one implementation manner. Fig.14 As shown, the electronic device includes a memory 140 for storing a computer program; and a processor 141 for implementing the steps of the cost map generation method mentioned in any of the above embodiments when executing the computer program.

[0142] Among them, the processor 141 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the processor 141 may also be a controller, a microcontroller, a microprocessor or other data processing chip. The processor 141 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 141 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 141 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 141 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0143] The memory 140 may include one or more computer non-volatile storage media, which may be non-transitory. The memory 140 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the memory 140 may be an internal storage unit of an electronic device, such as a hard disk of a server. In other embodiments, the memory 140 may also be an external storage device of an electronic device, such as a plug-in hard disk equipped on a server, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the memory 140 may also include both an internal storage unit of an electronic device and an external storage device. The memory 140 may not only be used to store application software and various types of data installed in the electronic device, such as: the code of the program in the process of executing the cost map generation method, but also be used to temporarily store data that has been output or is to be output. In this embodiment, the memory 140 is at least used to store the following computer program 1401, wherein, after the computer program is loaded and executed by the processor 141, the relevant steps of the cost map generation method disclosed in any of the aforementioned embodiments can be implemented. In addition, the resources stored in the memory 140 may also include an operating system 1402 and data 1403, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 1402 may include Windows, Unix, Linux, etc. The data 1403 may include but is not limited to data corresponding to the cost map generation result, etc.

[0144] In some embodiments, the electronic device may further include a display screen 142, an input / output interface 143, a communication interface 144 or a network interface, a power supply 145 and a communication bus 146. Among them, the display screen 142 and the input / output interface 143, such as a keyboard, belong to the user interface, and exemplary user interfaces may also include standard wired interfaces, wireless interfaces, and the like. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, and the like. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface. The communication interface 144 may exemplarily include a wired interface and / or a wireless interface, such as a WI-FI interface, a Bluetooth interface, and the like, which are generally used to establish a communication connection between the electronic device and other electronic devices. The communication bus 146 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, and the like. The bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Fig.14 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0145] Those skilled in the art will understand that Fig.14 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, for example, it may also include a sensor 147 to implement various functions.

[0146] It is understandable that if the cost map generation method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes but is not limited to: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, removable disk, CD-ROM, magnetic disk or optical disk, etc. Various media that can store program codes. Based on this, the present invention also provides a non-volatile storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the cost map generation method recorded in any of the above embodiments are executed.

[0147] It is understandable that if the cost map generation method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, the computer software product does not need to be stored in a physical storage medium. For example, it can be directly transmitted to a computer or other device with information processing capabilities through a wired network or a wireless network to perform all or part of the steps of the methods of various embodiments of the present invention. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product. Based on this, the present invention also provides a computer program product that stores a computer program. When the computer program is executed by a processor, the steps of the cost map generation method recorded in any of the above embodiments are performed.

[0148] Finally, the present invention also provides an unmanned vehicle that fully reuses its own sensors, such as cameras, lidars, odometers, and inertial measurement units, without the need to install other additional sensors. A computer program that implements the cost map generation method recorded in any of the previous embodiments is built into the existing autonomous driving perception system. The autonomous driving perception system can be used to implement the steps of the cost map generation method recorded in any of the previous embodiments when executing the computer program stored in the memory. Based on the image data and point cloud data collected by multi-source sensors, on the one hand, the spatial position information of common dynamic obstacles such as pedestrians and vehicles is obtained, and on the other hand, the three-dimensional position and morphology of low obstacles / potholes can be accurately identified, such as speed bumps and cracks, deep pits, shallow pits or puddles. At the same time, large-scale static obstacles such as walls, bushes, trees, etc. and ground terrain information can be obtained. Finally, multiple types of information are merged to construct a three-dimensional semantic-level cost map, such as Fig.15 As shown, low obstacle / pothole information is integrated into autonomous driving planning and control to enhance the vehicle's obstacle avoidance capability and effectively improve the safety and comfort of the autonomous driving system.

[0149] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the hardware disclosed in the embodiments, including electronic devices, non-volatile storage media, computer program products and unmanned vehicles, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0150] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0151] The above is a detailed introduction to a cost map generation method, electronic device, non-volatile storage medium, computer program product and unmanned vehicle provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that based on the embodiments of the present invention, for ordinary technicians in this technical field, all other embodiments obtained without creative work are within the scope of protection of the present invention. Without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A cost map generation method, characterized in that: include: Construct a blank 3D cost map based on the driving area to be detected; The to-be-detected driving area is rasterized into a bird's-eye view grid area including a plurality of grids, each grid including a spatial voxel in a horizontal direction perpendicular to the road surface, and the spatial voxel includes a plurality of cubes; The target detection algorithm is used to perform target recognition processing on the image data and point cloud data of the driving area to be detected, and the first type of obstacles and their three-dimensional spatial data are obtained; The three-dimensional spatial data is the spatial position and geometric shape data of the first type of obstacles; Target detection is performed on the image data of the to-be-detected driving area and the point graph fusion features of the point cloud data using a target detection algorithm to obtain a second type of obstacle and its three-dimensional shape information, and the three-dimensional shape information is updated according to each frame of point cloud data continuously acquired during the vehicle's forward movement; wherein the elevation information of the second type of obstacle is less than the elevation information of the first type of obstacle; and the three-dimensional shape information is the spatial position, elevation information and geometric shape data of the second type of obstacle; Fill the blank three-dimensional cost map according to the three-dimensional spatial data and the three-dimensional shape information to obtain a three-dimensional cost map; Wherein, updating the three-dimensional shape information according to each frame of point cloud data continuously acquired during the vehicle's forward movement includes: Determine the grids included in the obstacle area of ​​the current frame according to the position information of the obstacle identified in the current frame in the bird's-eye view grid area at the current moment; According to the new grid elevation information corresponding to the point cloud data in the obstacle area of ​​each frame continuously acquired during the vehicle's forward movement, the spatial voxel weight at the current moment is updated, and the elevation information and geometric morphology data of the grid in the obstacle area are updated based on the updated spatial voxel weight to continuously update the three-dimensional morphology information.

2. The costmap generation method according to claim 1, characterized in that: Before filling the blank three-dimensional cost map according to the three-dimensional spatial data and the three-dimensional shape information to obtain the three-dimensional cost map, the method further includes: Extracting ground point cloud data corresponding to the ground area from the point cloud data according to the ground area determined by the image data and calibration parameters between the laser radar and the camera; According to the clustering result of the average normal vectors of each grid in the ground area, the ground area is segmented to obtain a plurality of sub-ground areas; According to the point cloud data contained in each sub-ground area, a plane fitting process is performed on the corresponding sub-ground area to determine the ground elevation information and ground tilt information of each sub-ground area to obtain three-dimensional ground data as the ground of the three-dimensional cost map.

3. The costmap generation method according to claim 2, characterized in that: Filling the blank three-dimensional cost map according to the three-dimensional spatial data and the three-dimensional shape information to obtain a three-dimensional cost map includes: According to using the three-dimensional ground data as the three-dimensional ground layer of the three-dimensional cost map, filling the blank three-dimensional cost map to obtain an initial three-dimensional cost map; The three-dimensional spatial data and the three-dimensional shape information are added to the initial three-dimensional cost map to generate a three-dimensional cost map.

4. The costmap generation method according to claim 3, characterized in that: The first type of obstacles includes dynamic obstacles and static obstacles, and adding the three-dimensional space data and the three-dimensional shape information to the initial three-dimensional cost map includes: Adding the three-dimensional bounding box and posture information of the dynamic obstacle to the initial three-dimensional cost map; For each static obstacle, if the overlapping area of ​​the current static obstacle identification area and the dynamic obstacle identification area is greater than the preset area threshold, the three-dimensional position and occupied space data of the current static obstacle are not added; if the overlapping area of ​​the current static obstacle identification area and the dynamic obstacle identification area is less than or equal to the preset area threshold, the three-dimensional position and occupied space data of the current static obstacle are added to the initial three-dimensional cost map.

5. The costmap generation method according to claim 1, characterized in that: The first type of obstacle is a static obstacle. By performing target recognition processing on the image data and point cloud data of the driving area to be detected, the first type of obstacle and its three-dimensional spatial data are obtained, including: Extracting two-dimensional image features from the image data, and identifying a ground area contained in the two-dimensional image features; Extracting ground point cloud data corresponding to the ground area from the point cloud data according to the ground area and calibration parameters between the laser radar and the camera, and obtaining a ground plane according to a plane fitting result of the ground point cloud data; Taking the ground plane as a reference, taking the point cloud data in the point cloud data that is within a preset elevation threshold range as the projected point cloud data projected onto the ground; Removing the ground point cloud data from the projected point cloud data to obtain non-ground point cloud data; According to the clustering result of the non-ground point cloud data, multiple obstacle areas are obtained, and by fitting the point cloud data of each obstacle area, the three-dimensional position and occupied space data of the corresponding obstacle area are obtained as the three-dimensional space data of the static obstacle.

6. The costmap generation method according to claim 1, characterized in that: The first type of obstacle is a dynamic obstacle. By performing target recognition processing on the image data and point cloud data of the driving area to be detected, the first type of obstacle and its three-dimensional spatial data are obtained, including: Pre-train an encoder-decoder network model, where the encoder is used to fuse input features and the decoder is used to perform target recognition processing; Respectively converting the image data and the point cloud data of the driving area to be detected to the target perspective to obtain point cloud features and image features; Inputting the point cloud features and the image features into the encoder for fusion to obtain point image fusion features; The point image fusion features are input into the decoder to obtain the three-dimensional bounding box and posture angle of the dynamic obstacle as the three-dimensional spatial data of the dynamic obstacle.

7. The costmap generation method according to claim 1, characterized in that: The target detection algorithm is used to perform target detection on the image data of the driving area to be detected and the point graph fusion features of the point cloud data to obtain the second type of obstacles and their three-dimensional shape information, including: Pre-constructing and training a three-dimensional target detection network model; the three-dimensional target detection network model includes a detection task head, a classification task head and an instance segmentation task head, the detection task head is used to identify the second type of obstacles and output the second type of obstacle position information; the classification task head is used to identify the second type of obstacle category information; the instance segmentation task head is used to segment the second type of obstacles on the target plane to obtain an instance segmentation result; Inputting the point graph fusion features into the three-dimensional target detection network model to obtain the position information, category information and instance segmentation results of the second type of obstacles; Based on the instance segmentation result and the point cloud data in the obstacle area, determining the geometric morphology description data of the second type of obstacles through a three-dimensional surface fitting result; The position information, category information and geometric description data of the second type of obstacles are used as the three-dimensional shape information of the second type of obstacles.

8. The costmap generation method according to claim 1, characterized in that: According to the position information of the obstacle identified in the current frame in the bird's-eye view grid area at the current moment, the grids included in the obstacle area of ​​the current frame are determined, including: According to the second type obstacle recognition result corresponding to the current frame, determine the grid in the corresponding second type obstacle area; If the current point position is the position information of the second type of obstacle identified in the current frame in the bird's-eye view grid area at the current moment, a first preset value is set for the current point as a mask value; if the current point position is not the position information of the second type of obstacle identified in the current frame in the bird's-eye view grid area at the current moment, a second preset value is set for the current point as a mask value; Obstacle region mask information is generated according to the mask value of each point in the bird's-eye view grid region to identify the grids included in the obstacle region of the current frame.

9. The costmap generation method according to any one of claims 1 to 8, characterized in that: The three-dimensional shape information is updated according to each frame of point cloud data continuously acquired during the vehicle's forward movement, including: For each grid in the rasterized driving area to be detected, a target grid belonging to the second type of obstacle area identified in the current frame is obtained; If the current target grid does not belong to the radar point cloud projection area, determining the weight of the spatial voxel of the current target grid based on the spatial voxel coordinate position of each target grid in the second type of obstacle area; If the current target grid belongs to the radar point cloud projection area, the weight of the spatial voxel of the current target grid is determined based on the spatial voxel coordinate position of each target grid in the second obstacle area and the elevation information of the radar point cloud projection area.

10. The costmap generation method according to any one of claims 1 to 8, characterized in that: Update the elevation information of the grid in the obstacle area based on the updated spatial voxel weights, including: Pre-training an elevation classification network, the elevation classification network is used to convert the depth estimation problem into a classification problem, and the elevation classification network determines a loss function based on obstacle type, elevation truth encoding information, obstacle area mask information, point cloud features and image features of the second type obstacle area grid, and spatial voxel weights of the second type obstacle area grid; The elevation information of the grid in the second type of obstacle area is obtained through the elevation classification network.

11. An electronic device, characterized in that: It comprises a processor and a memory, wherein the processor is used to implement the steps of the cost map generation method as claimed in any one of claims 1 to 10 when executing the computer program stored in the memory.

12. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the cost map generation method according to any one of claims 1 to 10 are implemented.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the cost map generation method described in any one of claims 1 to 10 are implemented.

14. An unmanned vehicle, characterized in that: It includes an autonomous driving perception system, which is used to implement the steps of the cost map generation method as described in any one of claims 1 to 10 when executing a computer program stored in a memory.

Images