Automatic driving path planning method and device based on three-dimensional spatial semantic information

Abstract

The application discloses an automatic driving path planning method and device based on three-dimensional space semantic information, point clouds obtained through three-dimensional reconstruction of images are used to correct semantic segmentation results of two-dimensional images, and the corrected space semantic information is used to provide a semantic grid height map for path planning, so that three-dimensional path planning is completed. Main steps include: performing semantic segmentation on a two-dimensional image; completing three-dimensional reconstruction by using depth information, mapping the semantic segmentation result to a three-dimensional space, and generating point clouds with spatial position relations; correcting the semantic result by using the spatial position information, dividing a dangerous area, and generating a semantic grid height map; and completing path planning in the three-dimensional space. The application fully utilizes three-dimensional spatial position information and two-dimensional image semantic segmentation results, improves the accuracy of dangerous area identification, and provides sufficient guarantee for the safety of the path planning method.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving and autonomous navigation for unmanned vehicles, and in particular to an autonomous driving path planning method and apparatus based on three-dimensional spatial semantic information. Background Technology

[0002] Path planning is a crucial step in achieving autonomous navigation for self-driving vehicles. In simplified man-made scenarios, including indoor environments and straight roads, it's generally assumed that the traversable area lies on a single plane. Therefore, many existing path planning methods focus on outputting trajectory points in a two-dimensional plane. However, in outdoor scenarios, such as mountainous sections or hilly areas in western plateaus, there are no clear road edges to delineate safe traversable areas, and the assumption of a single-plane trajectory cannot be used. Furthermore, dangerous objects such as falling rocks and black ice may need to be identified on the road surface. Therefore, accurate hazard area delineation and three-dimensional path planning for self-driving vehicles are essential and practically significant research areas.

[0003] For 3D path planning, if the map only uses binary labels for passable and impassable paths, it cannot reflect complex environmental characteristics, and it cannot obtain sufficient information from the map to judge potential dangerous actions such as chassis getting stuck or tires being suspended in the air. Currently, widely used map representation methods include: raster methods, visibility maps, Vino maps, and topological methods. The raster method, as a mature mapping method used for planning, was initially proposed by W.E. Howden in 1968. Its principle is to segment the external environment image information into units and represent them using rectangular graticules of equal size with binary information. The size of the graticule is an important indicator affecting the robustness of the planning algorithm; smaller graticules result in higher environmental resolution, but also incur greater computational overhead. Improvements to raster maps include establishing a raster correlation matrix to calculate the probability of candidate graticules and using graticule direction vectors for path guidance, effectively improving path recognition capabilities and planning efficiency. The visibility map method converts obstacles in the external environment into convex polygons. It connects the vertices of the obstacle with the vehicle's current position and the target point using straight lines. If the line segment connecting two points does not intersect any obstacle polygon, then the two points are considered visible. This method can reduce map data storage costs and accelerate path deceleration efficiency, but it is difficult to extend to complex scenes. To address this issue, improvements to existing methods include: establishing a visibility map node judgment mechanism, requiring only the construction of visible edges with a high degree of matching with the optimal path, thereby improving the algorithm's operating efficiency; performing rule-based filtering of obstacles in path planning, excluding obstacles with low relevance, and reducing the number of candidate reference paths by simplifying the number of edges in the visibility map, thereby improving the algorithm's computational efficiency and reliability. The aforementioned existing methods utilize spatial information when constructing the environmental map for path planning, but do not incorporate semantic information from scene understanding.

[0004] With the continuous development of artificial intelligence and neural networks, semantic segmentation can label the category of each pixel in an image, providing a foundation for scene understanding. Existing semantic segmentation methods for 2D images can be summarized as block classification, fully convolutional network methods, and encoding / decoding architectures. Semantic segmentation techniques combined with depth information primarily combine features from both color and depth modalities as network input for training and inference. The problem with this approach is that both modalities inevitably contain noise, affecting the accuracy of semantic segmentation; furthermore, it fails to fully utilize spatial clustering relationships through 3D reconstruction. This invention first performs depth estimation on the image, then completes 3D reconstruction, uses spatial information relationships to correct the 2D semantic segmentation results, and generates a semantic raster height map, marking dangerous areas to complete 3D path planning. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing an autonomous driving path planning method and apparatus based on three-dimensional spatial semantic information.

[0006] The objective of this invention is achieved through the following technical solution: an autonomous driving path planning method based on three-dimensional spatial semantic information, comprising the following steps:

[0007] (1) Using any image x i As input, the image x is processed by a neural network. i Perform semantic segmentation to obtain image x i semantic segmentation results The semantic segmentation result Includes road boundaries, static obstacles, dynamic obstacles, and unknown areas; semantic segmentation results According to the hazard assessment criteria, the output is converted into image x. i semantic mask The hazard determination criterion is to assign a value of 1 to each pixel in the topologically connected region within the road boundary in the semantic segmentation result, and to assign a value of 0 to each pixel in the static obstacle, dynamic obstacle and unknown region in the semantic segmentation result.

[0008] (2) For any input image x i Depth estimation is performed to obtain image x i depth information set Using image x i depth information set and image x i Perform 3D reconstruction, and reconstruct the image x iEach pixel in the image is mapped to a corresponding spatial point in the world coordinate system, resulting in a 3D reconstructed point cloud. Any spatial point in the point cloud represents the x-axis of the image. i Any pixel in the image is converted into a point with a spatial positional relationship; then a semantic mask is used. Identify safe or dangerous points for each spatial point in the 3D reconstructed point cloud.

[0009] (3) Cluster the spatial points in the 3D reconstructed point cloud and use the spatial location information to correct the semantic mask. Obtain the corrected semantic mask The coordinates of each spatial point in the 3D reconstructed point cloud in the world coordinate system are converted to top-view coordinates to obtain a 2D map E. i Then, based on the corrected semantic mask... For two-dimensional map E i Rasterize the raster and assign raster semantics, then assign a corresponding raster height to each raster to obtain a semantic raster height map.

[0010] (4) Set the driving parameters for autonomous vehicles in the semantic grid height map. Q predicted trajectories are randomly generated, and it is determined whether a collision occurs on each predicted trajectory. If a collision occurs, the corresponding predicted trajectory is removed. From the remaining predicted trajectories after removal, the predicted trajectory with the shortest distance from the starting point to the ending point is selected as the safe path to enable autonomous vehicles to drive on the road.

[0011] Furthermore, step (2) specifically includes the following sub-steps:

[0012] (2.1) For any input image x i Depth estimation is performed to obtain image x i Each pixel {m1,m2,…,m a ,...,m N The corresponding depth information set Wherein, image x i There are N pixels in the CCP, a = 1, 2, ..., a, ..., N, d a For image x i any pixel m a The corresponding depth information; the depth information is the actual distance between each pixel in the two-dimensional image projected from the object in three-dimensional space and the camera;

[0013] (2.2) Take any image x i The coordinates of each pixel are transformed from the pixel coordinate system to the image coordinate system; the origin of the pixel coordinate system is O. uvFrom the origin O uv The right side of the image points to the u-axis of the pixel coordinate system, starting from the origin O. uv The v-axis points downwards in the image, which is the pixel coordinate system; the origin of the image coordinate system is O. xy From the origin O xy The x-axis of the pixel coordinate system points to the right side of the image, starting from the origin O. xy The y-axis points downwards in the image and is the pixel coordinate system; both the pixel coordinate system and the image coordinate system are Cartesian coordinate systems.

[0014] For image x i any pixel m a From the coordinates in the pixel coordinate system (u a ,v a Convert to coordinates in the image coordinate system (x) a ,y a The calculation formula for ) is as follows:

[0015]

[0016] Where u0 is the u-axis coordinate of the origin of the image coordinate system in the pixel coordinate system, and v0 is the v-axis coordinate of the origin of the image coordinate system in the pixel coordinate system; u a For pixel m a u-axis coordinates, v a For pixel m a v-axis coordinates; x a For pixel m a x-axis coordinates, y a For pixel m a The y-axis coordinate; dx is the resolution on the x-axis, in meters per pixel; dy is the resolution on the y-axis, in meters per pixel;

[0017] (2.3) Then the image x i The coordinates of each pixel are transformed from the image coordinate system to the camera coordinate system; the origin of the camera coordinate system is O. C From the origin O C The X-axis of the camera coordinate system points to the right of the camera. C The axis, from the origin O C The Y-axis pointing below the camera is the camera coordinate system. C The axis, from the origin O C The Z-axis pointing in front of the camera is the camera coordinate system. C axis;

[0018] For image x i any pixel m a From the coordinates (x) in the image coordinate system a ,y aConvert to coordinates in the image coordinate system (X) a ,Y a Z a The calculation formula for ) is as follows:

[0019]

[0020] Where f is the focal length of the camera; X a For pixel m a X C Y-axis coordinates a For pixel m a Y C Z-axis coordinates a For pixel m a Z C Axis coordinates;

[0021] (2.4) Then the image x i The coordinates of each pixel in the image are transformed from the camera coordinate system to the world coordinate system to obtain the 3D reconstructed point cloud {M}. i}: in, For image x i any pixel m a The corresponding spatial point in the camera coordinate system; the origin of the world coordinate system is O. W From the origin O W The right side pointing to the vehicle's initial position is the X-axis of the world coordinate system. W The axis, from the origin O W The direction pointing forward from the vehicle's initial position is the Y-axis of the world coordinate system. W The axis, from the origin O W The Z-axis, pointing and perpendicular to the Earth's surface, is the world coordinate system. W axis;

[0022] For image x i any pixel m a From the camera coordinate system coordinates (X a ,Y a Z a Convert to world coordinate system coordinates (X) a ′,Y a ′,Z a The formula for calculating ′) is as follows:

[0023]

[0024] Where R is the rotation matrix of the camera coordinate system relative to the world coordinate system, which is a 3×3 matrix; T is the translation vector of the camera coordinate system relative to the world coordinate system, which is a 3×1 vector; X a ′ represents pixel ma X W Y-axis coordinates; a ′ represents pixel m a Y W Z-axis coordinates; a ′ represents pixel m a Z W Axis coordinates;

[0025] (2.5) For any image x i For each pixel in the image, store any image x in the X channel. i X of each pixel W The x-axis coordinates are used to store the x-axis of any image in the Y-channel. i Y of each pixel W Axis coordinates, storing any image x in the Z channel i Z of each pixel W Axis coordinates, storing any semantic mask in I channels. Each pixel m a pixel value S a Then the pixel value S a A spatial point with a value of 1 is considered a safe point, and the pixel value S is set accordingly. a Points with a value of 0 are identified as danger points.

[0026] Furthermore, step (3) specifically includes the following sub-steps:

[0027] (3.1) From the 3D reconstructed point cloud {M i Randomly select a spatial point in} As the first clustering spatial point M1′, Euclidean clustering-based segmentation is adopted during clustering. The distance judgment criterion for Euclidean clustering segmentation is Euclidean distance. For the first clustering spatial point M1′, the nearest neighbor search algorithm is used to find K spatial points that are closest to the first clustering spatial point M1′. The K spatial points whose Euclidean distance from the first clustering spatial point M1′ is less than the Euclidean distance threshold are clustered into the point set U1 and then separated from the point cloud {M1′}. i The clustering process ends when the number of spatial points in point set U1 no longer increases, and the first clustered spatial point M1′ is obtained. The point set U1 after clustering is completed.

[0028] Then from the point cloud {M i Randomly select a spatial point from the remaining spatial points in the point cloud {M} as the first clustering spatial point M2′ and repeat the above steps until the point cloud {M} is filled in. i All spatial points in the array are clustered to obtain the clustered point set U1, ..., U2. j ...;

[0029] (3.2) For each point set U after clustering j Determine the point set U j The height difference between the edge of the point set and the nearest neighboring spatial points of other point sets is used to determine the point set U. When the height difference exceeds a height difference threshold, the point set U is removed. j All spatial points in the semantic mask Update the pixel values ​​in the point set U: assign a value of 0, otherwise update the pixel values ​​in the point set U. j All spatial points in the semantic mask The pixel values ​​in the image are not updated, resulting in the corrected semantic mask.

[0030] (3.3) Then the point cloud {M i Each spatial point in} In the world coordinate system, the coordinates (X) a ′,Y a ′,Z a Convert X'' to top view coordinates (X'') a ′,Y a ′), thus obtaining a two-dimensional map E from a top-down perspective. i The two-dimensional map contains the two-dimensional coordinates of each spatial point;

[0031] Then, based on the corrected semantic mask For two-dimensional map E i Rasterization is performed and raster semantics are assigned to obtain a semantic raster map. Define the raster resolution as D i The specific process of rasterization and assigning raster semantics is as follows: starting from the origin O of the camera coordinate system... C Begin, towards Y W Positive axis direction, per D i *D i Spatial points within an area are merged into a single grid. In any given grid, if the number of spatial points with a pixel value of 1 is greater than the number of spatial points with a pixel value of 0, the grid is considered a safe grid; otherwise, it is considered a dangerous grid.

[0032] Subsequently, semantic raster maps Z of all spatial points in each grid W The average of the axis coordinates is used as the raster height of the corresponding grid cell to obtain the semantic raster height map.

[0033] Furthermore, step (4) specifically includes the following sub-steps:

[0034] (4.1) When judging three-dimensional collisions, the autonomous vehicle G d Set it to a cuboid shape, and set the autonomous vehicle G... dDriving parameters: G of autonomous vehicle d The length is L d Width is W d And using the tire diameter as the height H d The collision expansion radius r and the passage height difference H are given. d / 2;

[0035] (4.2) Setting up the autonomous vehicle G d In semantic raster height map The vehicle G drives in the middle and randomly generates Q lines of autonomous driving vehicles G. d Predicted trajectory from start to finish: in, For any predicted trajectory; when randomly generating the predicted trajectory for autonomous vehicles, avoid the semantic raster height map. Any dangerous grid cell;

[0036] (4.3) For any predicted trajectory Set the predicted trajectory The path point at the current moment The coordinates of the top view in the world coordinate system are: in, For predicting trajectories The path point at the current moment X in the world coordinate system W Axis coordinates For predicting trajectories The path point at the current moment Y in the world coordinate system W Axis coordinates; path points at the next time step The coordinates of the top view in the world coordinate system are: in, For predicting trajectories The path point at the next time step X in the world coordinate system W Axis coordinates For predicting trajectories The path point at the next time step Y in the world coordinate system W Axis coordinates; G-axis coordinates for autonomous vehicles d Yaw angle at the current moment From path points Point to path Direction and X W Angle along the negative axis:

[0037]

[0038] autonomous vehicles Gd Located at the waypoint The coordinates of the centers of the four wheels (right rear, left rear, right front, and left front) in the top view of the world coordinate system are as follows: and It is calculated using the following formula:

[0039]

[0040] in, For autonomous vehicles G d Located at the waypoint The center of the right rear wheel is located in the X coordinate system at the current moment. W Axis coordinates;

[0041] For autonomous vehicles G d Located at the waypoint The center of the right rear wheel is located in the Y-axis of the world coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the left rear wheel is located in the X coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the left rear wheel is located in the Y-axis of the world coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the right front wheel is located in the X coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the right front wheel is located in the Y-axis of the world coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the left front wheel is located in the X coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the left front wheel is located in the Y-axis of the world coordinate system at the current moment. W Axis coordinates;

[0042] autonomous vehicle G d Located at the waypoint The coordinates of the centers of the four wheels (right rear, left rear, right front, and left front) in the world coordinate system are as follows: and in, For autonomous vehicles G d Located at the waypoint The grid height of the grid where the center of the right rear wheel is located. For autonomous driving vehicles G d Located at the waypoint The grid height of the grid where the center of the left rear wheel is located. For autonomous driving vehicles G d Located at the waypoint The grid height of the grid where the center of the right front wheel is located. For autonomous driving vehicles G d Located at the waypoint The grid height of the grid where the center of the left front wheel is located;

[0043] Using the least squares method combined with the G of autonomous vehicles d Located at the waypoint The coordinates of the centers of the four wheels (right rear, left rear, right front, and left front) in the world coordinate system are used to construct the G-axis coordinates of the autonomous vehicle. d The planar equation of the chassis:

[0044]

[0045] in, These are the coefficients of the plane equation;

[0046] Singular value decomposition of the leftmost matrix in the plane equation yields:

[0047]

[0048] in, and V t d,q Each of these is a 4×4 orthogonal matrix; Let T be a diagonal matrix composed of singular values, and let T be the matrix transpose operator;

[0049] The orthogonal matrix V is calculated. t d,q and the orthogonal matrix V t d,q The values ​​of the four elements in the last column correspond to and

[0050] (4.4) G will be an autonomous vehicle d Located at the waypoint The region where the distance from the center of the chassis is less than the collision expansion radius r is used as the path point. neighborhood Then, the neighborhoods are selected sequentially. For each grid cell, a height collision determination is performed: for the neighborhood... any grid Get the grid The center point in the top view coordinates of the world coordinate system in, For grid The center point in the world coordinate system X W Axis coordinates For grid The center point in the world coordinate system Y W Axis coordinates; when the grid The grid height is greater than Then determine the path point. neighborhood The inner chassis will collide with the grille, but the opposite will not.

[0051] (4.5) Repeat steps (4.3)-(4.4) for Q predicted trajectories. If the chassis collides with the grid in the neighborhood of any path point in the predicted trajectory, the predicted trajectory is removed.

[0052] (4.6) After the elimination is completed, if Q predicted trajectories are not completely eliminated, select the predicted trajectory with the shortest distance from the start point to the end point from the remaining predicted trajectories as the safe path to realize the safe path planning of the autonomous vehicle on the road; if Q predicted trajectories are completely eliminated, regenerate Q predicted trajectories randomly and repeat steps (4.3)-(4.5) until a safe path is selected to realize the safe path planning of the autonomous vehicle on the road; if a safe path is not selected after the selection process exceeds 10 minutes, exit the autonomous driving path planning and the operator will intervene manually.

[0053] The present invention also provides an autonomous driving path planning device based on three-dimensional spatial semantic information, including one or more processors, for implementing the above-mentioned autonomous driving path planning method based on three-dimensional spatial semantic information.

[0054] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-described autonomous driving path planning method based on three-dimensional spatial semantic information.

[0055] The beneficial effects of this invention are as follows: Compared with existing technologies, this invention significantly improves the accuracy of hazardous area delineation and the safety of 3D path planning. Firstly, while existing methods generally rely solely on spatial height information or scene semantic information to provide map references for path planning, this invention maps the 2D semantic segmentation results into 3D space, then eliminates mis-segmented outliers through identifier clustering, and supplements missing hazardous clusters based on height difference thresholds. This effectively improves the accuracy of hazardous area delineation. Secondly, a semantic raster height map is generated using point clouds with semantic identifiers. Combined with the index proposed in this invention for determining whether a 3D path will collide on the raster height map, path planning is completed, reducing the collision rate and improving the safety of the method. Attached Figure Description

[0056] Figure 1 This is a flowchart illustrating an autonomous driving path planning method based on three-dimensional spatial semantic information.

[0057] Figure 2 This is a schematic diagram showing any image, its corresponding semantic segmentation result, and its semantic mask. Figure 2 (a) is a schematic diagram of any image in pixel coordinates. Figure 2 (b) is a schematic diagram of the semantic segmentation results for any given image. Figure 2 (c) is a schematic diagram of the semantic mask corresponding to any image;

[0058] Figure 3 This is a schematic diagram of any image in pixel coordinates and image coordinates.

[0059] Figure 4 This is a schematic diagram illustrating the projection relationship between the camera coordinate system and the image coordinate system.

[0060] Figure 5 This is a schematic diagram of any predicted trajectory;

[0061] Figure 6 This is a schematic diagram of the structure of an autonomous driving path planning device based on three-dimensional spatial semantic information. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0063] This invention proposes an autonomous driving path planning method based on three-dimensional spatial semantic information. By fully utilizing three-dimensional spatial location information and two-dimensional image semantic segmentation results, it improves the accuracy of dangerous area identification and enhances the safety of three-dimensional path planning. The method mainly includes the following steps: 1) performing semantic segmentation on two-dimensional images using a neural network; 2) estimating the depth of the image, using depth information to complete three-dimensional reconstruction, mapping the semantic segmentation results to three-dimensional space, and generating a point cloud with spatial positional relationships; 3) correcting the two-dimensional image semantic segmentation results using the clustering results of the spatial point cloud, using semantic information and height difference thresholds to determine and delineate dangerous areas and generate a semantic raster height map; 4) completing path planning in three-dimensional space with collision avoidance as the safety criterion. This invention is applicable to rugged three-dimensional roads in the wild, enabling autonomous vehicles traveling on the ground to achieve safe path planning and navigation to their destination.

[0064] This invention includes the following three key technical features:

[0065] 1) Map the two-dimensional image to the three-dimensional space based on the depth estimation information, and store the semantic segmentation results as point cloud reflection intensity channels when mapping pixels one by one. Define each pixel category to correspond to discrete reflection intensity values, thereby completing the association between two-dimensional semantic information and three-dimensional spatial information.

[0066] 2) Utilize the spatial location information obtained from 3D reconstruction to correct the semantic segmentation results of the 2D image and jointly generate a semantic raster height map: reproject outliers that deviate from their semantic label clustering locations onto the 2D image plane and delete their erroneous labels; cluster point clouds with edge height differences greater than a threshold as dangerous areas, and combine these dangerous areas from the semantic labels to generate a semantic raster height map. The overall method flow is as follows: Figure 2 As shown.

[0067] 3) An innovative index for judging the safety of three-dimensional paths on a grid height map was designed to assist in path planning and reduce the occurrence of dangerous situations.

[0068] Example 1

[0069] like Figure 1 As shown, this invention provides an autonomous driving path planning method based on three-dimensional spatial semantic information, comprising the following steps:

[0070] (1) Using any image x i As input, the image x is processed by a neural network. i Perform semantic segmentation to obtain image x i semantic segmentation results The semantic segmentation result Includes road boundaries, static obstacles, dynamic obstacles, and unknown areas; semantic segmentation results According to the hazard assessment criteria, the output is converted into image x. i semantic mask The hazard determination criterion is as follows: Each pixel in the topologically connected region within the road boundary in the semantic segmentation result is assigned a value of 1, indicating a region that can be safely passed; while each pixel in static obstacles, dynamic obstacles, and unknown regions in the semantic segmentation result is assigned a value of 0, indicating a region where passage is dangerous. This results in a single-channel image with the same length and width as the single image, i.e., a semantic mask. Semantic mask The pixel value of each pixel in the middle is either 1 or 0.

[0071] The image x i For a two-dimensional image, the semantic mask To be related to image x i Single-channel images of the same size.

[0072] In this embodiment, any image x i like Figure 2 As shown in (a); Figure 2 In (a), the image x i The top left corner is taken as the origin O of the pixel coordinate system. uv From the origin O uv The right side of the image points to the u-axis of the pixel coordinate system, starting from the origin O. uv The v-axis points below the image in the pixel coordinate system; for any image x i Each pixel m a The coordinates in the pixel coordinate system are (u a ,v b ), where u a For pixel m a u-axis coordinates, v b For pixel m a v-axis coordinates; m for each pixel a In semantic mask The pixel value in S a Semantic segmentation results like Figure 2 As shown in (b). Semantic mask like Figure 2 As shown in (c).

[0073] (2) For any input image x i Depth estimation is performed to obtain image x i depth information set Using image x i depth information set and image x iPerform 3D reconstruction, and reconstruct the image x i Each pixel in the image is mapped to a corresponding spatial point in the world coordinate system, resulting in a 3D reconstructed point cloud. Any spatial point in the point cloud represents the x-axis of the image. i Any pixel in the image is converted into a point with a spatial positional relationship; then a semantic mask is used. Each spatial point in the 3D reconstructed point cloud is identified as a safe point or a dangerous point.

[0074] Step (2) specifically includes the following sub-steps:

[0075] (2.1) For any input image x i Depth estimation is performed to obtain image x i Each pixel {m1,m2,...,m a ,...,m N The corresponding depth information set Wherein, image x i There are N pixels in the CCP, a = 1, 2, ..., a, ..., N, d a For image x i any pixel m a The corresponding depth information; the depth information is the actual distance between each pixel in the two-dimensional image projected from the object in three-dimensional space and the camera. For images acquired by an RGB-D camera, depth information can be directly obtained using the depth image; for images acquired by a monocular camera, depth information can be obtained through triangulation or deep learning estimation.

[0076] (2.2) Take any image x i The coordinates of each pixel are transformed from the pixel coordinate system to the image coordinate system; the origin of the pixel coordinate system is O. uv From the origin O uv The right side of the image points to the u-axis of the pixel coordinate system, starting from the origin O. uv The v-axis points downwards in the image, which is the pixel coordinate system; the origin of the image coordinate system is O. xy From the origin O xy The x-axis of the pixel coordinate system points to the right side of the image, starting from the origin O. xy The y-axis points downwards in the image, representing the pixel coordinate system; both the pixel coordinate system and the image coordinate system are Cartesian coordinate systems. For any image x... i A schematic diagram of the pixel coordinate system and the image coordinate system is shown below. Figure 3 As shown.

[0077] For image x i any pixel m a From the coordinates in the pixel coordinate system (u a ,va Convert to coordinates in the image coordinate system (x) a ,y a The calculation formula for ) is as follows:

[0078]

[0079] Where u0 is the u-axis coordinate of the origin of the image coordinate system in the pixel coordinate system, and v0 is the v-axis coordinate of the origin of the image coordinate system in the pixel coordinate system; u a For pixel m a u-axis coordinates, v a For pixel m a v-axis coordinates; x a For pixel m a x-axis coordinates, y a For pixel m a The y-axis coordinate; dx is the resolution on the x-axis, in meters per pixel; dy is the resolution on the y-axis, in meters per pixel.

[0080] (2.3) Then the image x i The coordinates of each pixel are transformed from the image coordinate system to the camera coordinate system; the camera coordinate system is a three-dimensional spatial coordinate system, and the projection relationship between the camera coordinate system and the image coordinate system is as follows: Figure 4 As shown; the origin of the camera coordinate system is O. C From the origin O C The X-axis of the camera coordinate system points to the right of the camera. C The axis, from the origin O C The Y-axis pointing below the camera is the camera coordinate system. C The axis, from the origin O C The Z-axis pointing in front of the camera is the camera coordinate system. C axis;

[0081] For image x i any pixel m a From the coordinates (x) in the image coordinate system a ,y a Convert to coordinates in the image coordinate system (X) a ,Y a Z a The calculation formula for ) is as follows:

[0082]

[0083] Where f is the focal length of the camera; X a For pixel m a X C Y-axis coordinates a For pixel m a Y CZ-axis coordinates a For pixel m a Z C Axis coordinates. Three coordinate axes, X, ... C Axis, Y C Axis and Z C The axes are in a rectangular coordinate system that follows the right-hand rule.

[0084] (2.4) Then the image x i The coordinates of each pixel in the image are transformed from the camera coordinate system to the world coordinate system to obtain the 3D reconstructed point cloud {M}. i}: in, For image x i any pixel m a The corresponding spatial point in the camera coordinate system; the origin of the world coordinate system is O. W From the origin O W The right side pointing to the vehicle's initial position is the X-axis of the world coordinate system. W The axis, from the origin O W The direction pointing forward from the vehicle's initial position is the Y-axis of the world coordinate system. W The axis, from the origin O W The Z-axis, pointing and perpendicular to the Earth's surface, is the world coordinate system. W axis.

[0085] For image x i any pixel m a From the camera coordinate system coordinates (X a ,Y a Z a Convert to world coordinate system coordinates (X) a ′,Y a ′,Z a The formula for calculating ′) is as follows:

[0086]

[0087] Where R is the rotation matrix of the camera coordinate system relative to the world coordinate system, which is a 3×3 matrix; T is the translation vector of the camera coordinate system relative to the world coordinate system, which is a 3×1 vector; X a ′ represents pixel m a X W Y-axis coordinates; a ′ represents pixel m a Y W Z-axis coordinates; a ′ represents pixel m a Z W Axis coordinates.

[0088] (2.5) For any image xi For each pixel in the image, store any image x in the X channel. i X of each pixel W The x-axis coordinates are used to store the x-axis of any image in the Y-channel. i Y of each pixel W Axis coordinates, storing any image x in the Z channel i Z of each pixel W Axis coordinates, storing any semantic mask in I channels. Each pixel m a pixel value S a This completes the association between two-dimensional semantic information and three-dimensional spatial information; then the pixel value S a A spatial point with a value of 1 is considered a safe point, and the pixel value S is set accordingly. a Points with a value of 0 are identified as danger points.

[0089] (3) Cluster the spatial points in the 3D reconstructed point cloud and use the spatial location information to correct the semantic mask. Obtain the corrected semantic mask The coordinates of each spatial point in the 3D reconstructed point cloud in the world coordinate system are converted to top-view coordinates to obtain a 2D map E. i Then, based on the corrected semantic mask... For two-dimensional map E i Rasterize the raster and assign raster semantics, then assign a corresponding raster height to each raster to obtain a semantic raster height map.

[0090] Step (3) specifically includes the following sub-steps:

[0091] (3.1) From the 3D reconstructed point cloud {M i Randomly select a spatial point in} As the first clustering spatial point M1′, Euclidean clustering-based segmentation is adopted during clustering. The distance judgment criterion for Euclidean clustering segmentation is Euclidean distance. For the first clustering spatial point M1′, the nearest neighbor search algorithm is used to find K spatial points that are closest to the first clustering spatial point M1′. The K spatial points whose Euclidean distance from the first clustering spatial point M1′ is less than the Euclidean distance threshold are clustered into the point set U1 and then separated from the point cloud {M1′}. i The clustering process continues until the number of spatial points in point set U1 no longer increases, resulting in the first clustered spatial point M1′ and the completed point set U1. In this embodiment, the Euclidean distance threshold is 0.3 meters.

[0092] Then from the point cloud {M iRandomly select a spatial point from the remaining spatial points in the point cloud {M} as the first clustering spatial point M2′ and repeat the above steps until the point cloud {M} is filled in. i All spatial points in} are clustered to obtain the point set U1, ..., Uj, ... after clustering.

[0093] (3.2) For each point set U after clustering j Determine the point set U j The height difference between the edge of the point set and the nearest neighboring spatial points of other point sets is used to determine the point set U. When the height difference exceeds a height difference threshold, the point set U is removed. j All spatial points in the semantic mask Update the pixel values ​​in the point set U: assign a value of 0, otherwise update the pixel values ​​in the point set U. j All spatial points in the semantic mask The pixel values ​​in the image are not updated, resulting in the corrected semantic mask. In this way, even if a certain danger category has not appeared in semantic segmentation, it can still be identified as a dangerous area and avoided before path planning; in this embodiment, the height difference threshold is 0.5 meters.

[0094] (3.3) Then the point cloud {M i Each spatial point in} In the world coordinate system, the coordinates (X) a ′,Y a ′,Z a Convert X'' to top view coordinates (X'') a ′,Y a ′), thus obtaining a two-dimensional map E from a top-down perspective. i The two-dimensional map contains the two-dimensional coordinates of each spatial point, that is, its height information is temporarily removed, and only the X coordinates of the world coordinate system are retained. W Y W Two-axis coordinates, in a top-down view of a two-dimensional map, are (X... a ′,Y a ′).

[0095] Then, based on the corrected semantic mask For two-dimensional map E i Rasterization is performed and raster semantics are assigned to obtain a semantic raster map. Define the raster resolution as D i The specific process of rasterization and assigning raster semantics is as follows: starting from the origin O of the camera coordinate system... C Begin, towards Y W Positive axis direction, per D i *D iSpatial points within an area are merged into a single grid. In any given grid, if the number of spatial points with a pixel value of 1 is greater than the number of spatial points with a pixel value of 0, the grid is considered a safe grid; otherwise, it is considered a dangerous grid.

[0096] Subsequently, semantic raster maps Z of all spatial points in each grid W The average of the axis coordinates is used as the raster height of the corresponding grid cell to obtain the semantic raster height map.

[0097] (4) Set the driving parameters for autonomous vehicles in the semantic grid height map. Q predicted trajectories are randomly generated, and it is determined whether a collision occurs on each predicted trajectory. If a collision occurs, the corresponding predicted trajectory is removed. From the remaining predicted trajectories after removal, the predicted trajectory with the shortest distance from the starting point to the ending point is selected as the safe path to enable autonomous vehicles to drive on the road.

[0098] Step (4) specifically includes the following sub-steps:

[0099] (4.1) When judging three-dimensional collisions, the autonomous vehicle G d Set it to a cuboid shape, and set the autonomous vehicle G... d Driving parameters: G of autonomous vehicle d The length is L d Width is W d And using the tire diameter as the height H d The collision expansion radius r and the passage height difference H are given. d / 2.

[0100] In this embodiment, r = 0.3m; assuming the autonomous vehicle G... d The chassis is located on the plane where the center of the tires lies, therefore the permissible height difference is set to H. d / 2.

[0101] (4.2) Setting up the autonomous vehicle G d In semantic raster height map The vehicle G drives in the middle and randomly generates Q lines of autonomous driving vehicles G. d Predicted trajectory from start to finish: in, For any predicted trajectory; when randomly generating the predicted trajectory for autonomous vehicles, avoid the semantic raster height map. Any dangerous grid cell. Any predicted trajectory. like Figure 5 As shown.

[0102] (4.3) For any predicted trajectory Set the predicted trajectory The path point at the current moment The coordinates of the top view in the world coordinate system are: in, For predicting trajectories The path point at the current moment X in the world coordinate system W Axis coordinates For predicting trajectories The path point at the current moment Y in the world coordinate system W Axis coordinates; path points at the next time step The coordinates of the top view in the world coordinate system are: in, For predicting trajectories The path point at the next time step X in the world coordinate system W Axis coordinates For predicting trajectories The path point at the next time step Y in the world coordinate system W Axis coordinates; G-axis coordinates for autonomous vehicles d Yaw angle at the current moment From path points Point to path Direction and X W Angle along the negative axis: Due to the autonomous vehicle G d Abstracted as a cuboid, its four vertices are considered to be the positions of the wheels.

[0103] autonomous vehicles G d Located at the waypoint The coordinates of the centers of the four wheels (right rear, left rear, right front, and left front) in the top view of the world coordinate system are as follows: and It is calculated using the following formula:

[0104]

[0105] in, For autonomous vehicles G d Located at the waypoint The center of the right rear wheel is located in the X coordinate system at the current moment. W Axis coordinates;

[0106] For autonomous vehicles G d Located at the waypoint The center of the right rear wheel is located in the Y-axis of the world coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the left rear wheel is located in the X coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the left rear wheel is located in the Y-axis of the world coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the right front wheel is located in the X coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the right front wheel is located in the Y-axis of the world coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the left front wheel is located in the X coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the left front wheel is located in the Y-axis of the world coordinate system at the current moment. W Axis coordinates.

[0107] autonomous vehicle G d Located at the waypoint The coordinates of the centers of the four wheels (right rear, left rear, right front, and left front) in the world coordinate system are as follows: and in, For autonomous vehicles G d Located at the waypoint The grid height of the grid where the center of the right rear wheel is located. For autonomous driving vehicles G d Located at the waypoint The grid height of the grid where the center of the left rear wheel is located. For autonomous driving vehicles G d Located at the waypoint The grid height of the grid where the center of the right front wheel is located. For autonomous driving vehicles G d Located at the waypoint The grid height of the grid where the center of the left front wheel is located.

[0108] Using the least squares method combined with the G of autonomous vehicles d Located at the waypoint The coordinates of the centers of the four wheels (right rear, left rear, right front, and left front) in the world coordinate system are used to construct the G-axis coordinates of the autonomous vehicle. d The planar equation of the chassis:

[0109]

[0110] in, These are the coefficients of the plane equation.

[0111] Singular value decomposition of the leftmost matrix in the plane equation yields:

[0112]

[0113] in, and V t d,qT Each of these is a 4×4 orthogonal matrix; It is a diagonal matrix composed of singular values.

[0114] The orthogonal matrix V is calculated. t d,q and the orthogonal matrix V t d,q The values ​​of the four elements in the last column correspond to and

[0115] (4.4) G will be an autonomous vehicle d Located at the waypoint The region where the distance from the center of the chassis is less than the collision expansion radius r is used as the path point. neighborhood Then, the neighborhoods are selected sequentially. For each grid cell, a height collision determination is performed: for the neighborhood... any grid Get the grid The center point in the top view coordinates of the world coordinate system in, For grid The center point in the world coordinate system X W Axis coordinates For grid The center point in the world coordinate system Y W Axis coordinates; when the grid The grid height is greater than Then determine the path point. neighborhood The inner chassis will collide with the grille, but the opposite will not.

[0116] (4.5) Repeat steps (4.3)-(4.4) for Q predicted trajectories. If the chassis collides with the grid in the neighborhood of any path point in the predicted trajectory, the predicted trajectory is removed.

[0117] (4.6) After the elimination is completed, if Q predicted trajectories are not completely eliminated, select the predicted trajectory with the shortest distance from the start point to the end point from the remaining predicted trajectories as the safe path to realize the safe path planning of the autonomous vehicle on the road; if Q predicted trajectories are completely eliminated, regenerate Q predicted trajectories randomly and repeat steps (4.3)-(4.5) until a safe path is selected to realize the safe path planning of the autonomous vehicle on the road; if a safe path is not selected after the selection process exceeds 10 minutes, exit the autonomous driving path planning and the operator will intervene manually.

[0118] Example 2

[0119] Corresponding to Embodiment 1 of the aforementioned autonomous driving path planning method based on three-dimensional spatial semantic information, the present invention also provides an embodiment of an autonomous driving path planning device based on three-dimensional spatial semantic information.

[0120] See Figure 6 The present invention provides an autonomous driving path planning device based on three-dimensional spatial semantic information, which includes one or more processors for implementing an autonomous driving path planning method based on three-dimensional spatial semantic information as described in the above embodiments.

[0121] An embodiment of the autonomous driving path planning device based on three-dimensional spatial semantic information of the present invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 6 The diagram shown is a hardware structure diagram of any data processing-capable device, including the autonomous driving path planning device based on three-dimensional spatial semantic information according to the present invention. (Except for...) Figure 6In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0122] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0123] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0124] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements an autonomous driving path planning method based on three-dimensional spatial semantic information as described in the preceding embodiments. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data-processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0125] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. An autonomous driving path planning method based on three-dimensional spatial semantic information, characterized in that, Includes the following steps: (1) Using any image x i As input, the image x is processed by a neural network. i Perform semantic segmentation to obtain image x i semantic segmentation results The semantic segmentation result Includes road boundaries, static obstacles, dynamic obstacles, and unknown areas; semantic segmentation results According to the hazard assessment criteria, the output is converted into image x. i semantic mask The hazard determination criterion is to assign a value of 1 to each pixel in the topologically connected region within the road boundary in the semantic segmentation result, and to assign a value of 0 to each pixel in the static obstacle, dynamic obstacle and unknown region in the semantic segmentation result. (2) For any input image x i Depth estimation is performed to obtain image x i depth information set Using image x i depth information set and image x i Perform 3D reconstruction, and reconstruct the image x i Each pixel in the image is mapped to a corresponding spatial point in the world coordinate system, resulting in a 3D reconstructed point cloud. Any spatial point in the point cloud represents the x-axis of the image. i Any pixel in the image is converted into a point with a spatial positional relationship; then a semantic mask is used. Identify safe or dangerous points for each spatial point in the 3D reconstructed point cloud. (3) Cluster the spatial points in the 3D reconstructed point cloud and use the spatial location information to correct the semantic mask. Obtain the corrected semantic mask The coordinates of each spatial point in the 3D reconstructed point cloud in the world coordinate system are converted to top-view coordinates to obtain a 2D map E. i Then, based on the corrected semantic mask... For two-dimensional map E i Rasterize the raster and assign raster semantics, then assign a corresponding raster height to each raster to obtain a semantic raster height map. (4) Set the driving parameters for autonomous vehicles in the semantic grid height map. Q predicted trajectories are randomly generated, and it is determined whether a collision occurs on each predicted trajectory. If a collision occurs, the corresponding predicted trajectory is removed. From the remaining predicted trajectories after removal, the predicted trajectory with the shortest distance from the starting point to the ending point is selected as the safe path to enable autonomous vehicles to drive on the road.

2. The autonomous driving path planning method based on three-dimensional spatial semantic information according to claim 1, characterized in that, Step (2) specifically includes the following sub-steps: (2.1) For any input image x i Depth estimation is performed to obtain image x i Each pixel {m1,m2,...,m a ,...,m N The corresponding depth information set Wherein, image x i There are N pixels in the CCP, a = 1, 2, ..., a, ..., N, d a For image x i any pixel m a The corresponding depth information; the depth information is the actual distance between each pixel in the two-dimensional image projected from the object in three-dimensional space and the camera; (2.2) Take any image x i The coordinates of each pixel are transformed from the pixel coordinate system to the image coordinate system; the origin of the pixel coordinate system is O. uv From the origin O uv The right side of the image points to the u-axis of the pixel coordinate system, starting from the origin O. uv The v-axis points downwards in the image, which is the pixel coordinate system; the origin of the image coordinate system is O. xy From the origin O xy The x-axis of the pixel coordinate system points to the right side of the image, starting from the origin O. xy The y-axis points downwards in the image and is the pixel coordinate system; both the pixel coordinate system and the image coordinate system are Cartesian coordinate systems. For image x i any pixel m a From the coordinates in the pixel coordinate system (u a ,v a Convert to coordinates in the image coordinate system (x) a ,y a The calculation formula for ) is as follows: Where u0 is the u-axis coordinate of the origin of the image coordinate system in the pixel coordinate system, and v0 is the v-axis coordinate of the origin of the image coordinate system in the pixel coordinate system; u a For pixel m a u-axis coordinates, v a For pixel m a v-axis coordinates; x a For pixel m a x-axis coordinates, y a For pixel m a The y-axis coordinate; dx is the resolution on the x-axis, in meters per pixel; dy is the resolution on the y-axis, in meters per pixel; (2.3) Then the image x i The coordinates of each pixel are transformed from the image coordinate system to the camera coordinate system; the origin of the camera coordinate system is O. C From the origin O C The X-axis of the camera coordinate system points to the right of the camera. C The axis, from the origin O C The Y-axis pointing below the camera is the camera coordinate system. C The axis, from the origin O C The Z-axis pointing in front of the camera is the camera coordinate system. C axis; For image x i any pixel m a From the coordinates (x) in the image coordinate system a ,y a Convert to coordinates in the image coordinate system (X) a ,Y a Z a The calculation formula for ) is as follows: Where f is the focal length of the camera; X a For pixel m a X C Y-axis coordinates a For pixel m a Y C Z-axis coordinates a For pixel m a Z C Axis coordinates; (2.4) Then the image x i The coordinates of each pixel in the image are transformed from the camera coordinate system to the world coordinate system to obtain the 3D reconstructed point cloud {M}. i }: in, For image x i any pixel m a The corresponding spatial point in the camera coordinate system; the origin of the world coordinate system is O. W From the origin O W The right side pointing to the vehicle's initial position is the X-axis of the world coordinate system. W The axis, from the origin O W The direction pointing forward from the vehicle's initial position is the Y-axis of the world coordinate system. W The axis, from the origin O W The Z-axis, pointing and perpendicular to the Earth's surface, is the world coordinate system. W axis; For image x i any pixel m a From the camera coordinate system coordinates (X a ,Y a Z a Convert to world coordinates (X′) a ,Y′ a ,Z′ a The calculation formula for ) is as follows: Where R is the rotation matrix of the camera coordinate system relative to the world coordinate system, which is a 3×3 matrix; T is the translation vector of the camera coordinate system relative to the world coordinate system, which is a 3×1 vector; X′ a For pixel m a X W Y-axis coordinates; a ′ represents pixel m a Y W Axis coordinates; Z′ a For pixel m a Z W Axis coordinates; (2.5) For any image x i For each pixel in the image, store any image x in the X channel. i X of each pixel W The x-axis coordinates are used to store the x-axis of any image in the Y-channel. i Y of each pixel W Axis coordinates, storing any image x in the Z channel i Z of each pixel W Axis coordinates, storing any semantic mask in I channels. Each pixel m a pixel value S a Then the pixel value S a A spatial point with a value of 1 is considered a safe point, and the pixel value S is set accordingly. a Points with a value of 0 are identified as danger points.

3. The autonomous driving path planning method based on three-dimensional spatial semantic information according to claim 2, characterized in that, Step (3) specifically includes the following sub-steps: (3.1) From the 3D reconstructed point cloud {M i Randomly select a spatial point in} As the first clustering point M′1, Euclidean clustering-based segmentation is adopted during clustering. The distance criterion for Euclidean clustering segmentation is Euclidean distance. For the first clustering point M′1, the nearest neighbor search algorithm is used to find K spatial points that are closest to the first clustering point M′1. These K spatial points whose Euclidean distance from the first clustering point M′1 is less than the Euclidean distance threshold are clustered into the point set U1 and then separated from the point cloud {M i The clustering process ends when the number of spatial points in point set U1 no longer increases, and the first clustered spatial point M′1 is obtained. The point set U1 after clustering is completed. Then from the point cloud {M i Randomly select a spatial point from the remaining spatial points in the point cloud {M} as the first clustering spatial point M′2 and repeat the above steps until the point cloud {M} is filled in. i All spatial points in the array are clustered to obtain the clustered point set U1, ..., U2. j ...; (3.2) For each point set U after clustering j Determine the point set U j The height difference between the edge of the point set and the nearest neighboring spatial points of other point sets is used to determine the point set U. When the height difference exceeds a height difference threshold, the point set U is removed. j All spatial points in the semantic mask Update the pixel values ​​in the point set U: assign a value of 0, otherwise update the pixel values ​​in the point set U. j All spatial points in the semantic mask The pixel values ​​in the image are not updated, resulting in the corrected semantic mask. (3.3) Then the point cloud {M i Each spatial point in} In the world coordinate system, the coordinates (X′) a ,Y′ a ,Z′ a Convert to top view coordinates (X′) a ,Y′ a This yields a 2D map E from a top-down perspective. i The two-dimensional map contains the two-dimensional coordinates of each spatial point; Then, based on the corrected semantic mask For two-dimensional map E i Rasterization is performed and raster semantics are assigned to obtain a semantic raster map. Define the raster resolution as D i The specific process of rasterization and assigning raster semantics is as follows: starting from the origin O of the camera coordinate system... C Begin, towards Y W Positive axis direction, per D i *D i Spatial points within an area are merged into a single grid. In any given grid, if the number of spatial points with a pixel value of 1 is greater than the number of spatial points with a pixel value of 0, the grid is considered a safe grid; otherwise, it is considered a dangerous grid. Subsequently, semantic raster maps were analyzed. Z of all spatial points in each grid W The average of the axis coordinates is used as the raster height of the corresponding grid cell to obtain the semantic raster height map.

4. The autonomous driving path planning method based on three-dimensional spatial semantic information according to claim 3, characterized in that, Step (4) specifically includes the following sub-steps: (4.1) When judging three-dimensional collisions, the autonomous vehicle G d Set it to a cuboid shape, and set the autonomous vehicle G... d Driving parameters: G of autonomous vehicle d The length is L d Width is W d And using the tire diameter as the height H d The collision expansion radius r and the passage height difference H are given. d / 2; (4.2) Setting up the autonomous vehicle G d In semantic grid height map The vehicle G drives in the middle and randomly generates Q lines of autonomous driving vehicles G. d Predicted trajectory from start to finish: in, For any predicted trajectory; when randomly generating the predicted trajectory for autonomous vehicles, avoid the semantic raster height map. Any dangerous grid cell; (4.3) For any predicted trajectory Set the predicted trajectory The path point at the current moment The coordinates of the top view in the world coordinate system are: in, For predicting trajectories The path point at the current moment X in the world coordinate system W Axis coordinates For predicting trajectories The path point at the current moment Y in the world coordinate system W Axis coordinates; path points at the next time step The coordinates of the top view in the world coordinate system are: in, For predicting trajectories The path point at the next time step X in the world coordinate system W Axis coordinates For predicting trajectories The path point at the next time step Y in the world coordinate system W Axis coordinates; G-axis coordinates for autonomous vehicles d Yaw angle at the current moment From path points Point to path Direction and X W Angle along the negative axis: autonomous vehicles G d Located at the waypoint The coordinates of the centers of the four wheels (right rear, left rear, right front, and left front) in the top view of the world coordinate system are as follows: and It is calculated using the following formula: in, For autonomous vehicles G d Located at the waypoint The center of the right rear wheel is located in the X coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the right rear wheel is located in the Y-axis of the world coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the left rear wheel is located in the X coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the left rear wheel is located in the Y-axis of the world coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the right front wheel is located in the X coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the right front wheel is located in the Y-axis of the world coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the left front wheel is located in the X coordinate system at the current moment. W Axis coordinates; For autonomous vehicles G d Located at the waypoint The center of the left front wheel is located in the Y-axis of the world coordinate system at the current moment. W Axis coordinates; autonomous vehicle G d Located at the waypoint The coordinates of the centers of the four wheels (right rear, left rear, right front, and left front) in the world coordinate system are as follows: and in, For autonomous vehicles G d Located at the waypoint The grid height of the grid where the center of the right rear wheel is located. For autonomous driving vehicles G d Located at the waypoint The grid height of the grid where the center of the left rear wheel is located. For autonomous driving vehicles G d Located at the waypoint The grid height of the grid where the center of the right front wheel is located. For autonomous driving vehicles G d Located at the waypoint The grid height of the grid where the center of the left front wheel is located; Using the least squares method combined with the G of autonomous vehicles d Located at the waypoint The coordinates of the centers of the four wheels (right rear, left rear, right front, and left front) in the world coordinate system are used to construct the G-axis coordinates of the autonomous vehicle. d The planar equation of the chassis: in, These are the coefficients of the plane equation; Singular value decomposition of the leftmost matrix in the plane equation yields: in, and V t d,q Each of these is a 4×4 orthogonal matrix; Let T be a diagonal matrix composed of singular values, and let T be the matrix transpose operator; The orthogonal matrix V is calculated. t d,q and the orthogonal matrix V t d,q The values ​​of the four elements in the last column correspond to and (4.4) G will be an autonomous vehicle d Located at the waypoint The region where the distance from the center of the chassis is less than the collision expansion radius r is used as the path point. neighborhood Then, the neighborhoods are selected sequentially. For each grid cell, a height collision determination is performed: for the neighborhood... any grid Get the grid The center point in the top view coordinates of the world coordinate system in, For grid The center point in the world coordinate system X W Axis coordinates For grid The center point in the world coordinate system Y W Axis coordinates; when the grid The grid height is greater than Then determine the path point. neighborhood The inner chassis will collide with the grille, but the opposite will not. (4.5) Repeat steps (4.3)-(4.4) for Q predicted trajectories. If the chassis collides with the grid in the neighborhood of any path point in the predicted trajectory, the predicted trajectory is removed. (4.6) After the elimination is completed, if Q predicted trajectories are not completely eliminated, select the predicted trajectory with the shortest distance from the start point to the end point from the remaining predicted trajectories as the safe path to realize the safe path planning of the autonomous vehicle on the road; if Q predicted trajectories are completely eliminated, regenerate Q predicted trajectories randomly and repeat steps (4.3)-(4.5) until a safe path is selected to realize the safe path planning of the autonomous vehicle on the road; if a safe path is not selected after the selection process exceeds 10 minutes, exit the autonomous driving path planning and the operator will intervene manually.

5. An autonomous driving path planning device based on three-dimensional spatial semantic information, characterized in that, It includes one or more processors for implementing the autonomous driving path planning method based on three-dimensional spatial semantic information as described in any one of claims 1-4.

6. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program is used to implement the autonomous driving path planning method based on three-dimensional spatial semantic information as described in any one of claims 1-4.

Images