Path planning method and device, computer readable storage medium and electronic equipment

Abstract

The invention belongs to the technical field of automatic driving, and particularly relates to a path planning method and device, a computer readable storage medium and electronic equipment. The method comprises the following steps: acquiring a multi-view camera image and a laser radar point cloud of a vehicle; determining aerial view perception information of the vehicle according to the multi-view camera image and the laser radar point cloud; processing the aerial view perception information through a track generation module to obtain a plurality of candidate path tracks of the vehicle; processing the bird's-eye view perception information and the plurality of candidate path trajectories through a driving area prediction module to obtain a future driving area of the vehicle; processing the bird's-eye view perception information and the plurality of candidate path tracks through an environment exploration module to obtain a future driving environment of the vehicle; and processing the future driving area and the future driving environment through a path scoring module to obtain a scoring result of the plurality of candidate path trajectories.

Description

Technical Field

[0001] The present application belongs to the field of autonomous driving technology, and in particular, relates to a path planning method, device, computer-readable storage medium, and electronic device. Background Art

[0002] In the field of autonomous driving technology, existing path planning methods are highly dependent on high-precision maps. However, the construction and maintenance costs of high-precision maps are very high, and high-precision maps may not be available in special scenarios (such as underground garages, remote areas, and private areas). Therefore, a path planning method that does not rely on high-precision maps is urgently needed. Summary of the invention

[0003] In view of this, embodiments of the present application provide a path planning method, device, computer-readable storage medium and electronic device to solve the problem of over-reliance on high-precision maps in existing path planning methods.

[0004] A first aspect of an embodiment of the present application provides a path planning method, which may include:

[0005] Acquire multi-view camera images and lidar point clouds of the vehicle;

[0006] Determining bird's-eye view perception information of the vehicle based on the multi-view camera image and the lidar point cloud;

[0007] Processing the bird's-eye view perception information through a trajectory generation module to obtain multiple candidate path trajectories of the vehicle;

[0008] Processing the bird's-eye view perception information and the plurality of candidate path trajectories through a driving area prediction module to obtain a future driving area of ​​the vehicle;

[0009] Processing the bird's-eye view perception information and the plurality of candidate path trajectories through an environment exploration module to obtain a future driving environment of the vehicle;

[0010] The future driving area and the future driving environment are processed by a path scoring module to obtain scoring results of the multiple candidate path trajectories.

[0011] In a specific implementation of the first aspect, determining the bird's-eye view perception information of the vehicle according to the multi-view camera image and the lidar point cloud may include:

[0012] Performing feature extraction on the multi-view camera image to obtain camera image features of the vehicle;

[0013] Performing feature extraction on the laser radar point cloud to obtain laser radar point cloud features of the vehicle;

[0014] Performing a bird's-eye view conversion on the camera image features to obtain a camera image bird's-eye view feature of the vehicle;

[0015] Performing a bird's-eye view conversion on the laser radar point cloud features to obtain a bird's-eye view feature of the laser radar point cloud of the vehicle;

[0016] Performing feature fusion on the bird's-eye view features of the camera image and the bird's-eye view features of the laser radar point cloud to obtain fused features of the vehicle;

[0017] Decoding the fusion features of the vehicle to obtain a bird's-eye view map and target detection information of the vehicle;

[0018] The bird's-eye view perception information of the vehicle is determined based on the bird's-eye view map and the target detection information.

[0019] In a specific implementation of the first aspect, the processing of the bird's-eye view perception information by a trajectory generation module to obtain a plurality of candidate path trajectories of the vehicle may include:

[0020] Generating a plurality of candidate path heat maps according to the bird's-eye view perception information; wherein each candidate path heat map is used to characterize the probability of each grid being a path point;

[0021] According to the probability of each grid being a path point, path points are extracted in the multiple candidate path heat maps to obtain multiple candidate path trajectories of the vehicle.

[0022] In a specific implementation of the first aspect, the driving area prediction module includes a spatial gated loop unit;

[0023] The processing of the bird's-eye view perception information and the plurality of candidate path trajectories by a driving area prediction module to obtain a future driving area of ​​the vehicle may include:

[0024] The spatial gated recurrent unit is used to iteratively process the bird's-eye view perception information and the multiple candidate path trajectories step by step in time to obtain the future driving area of ​​the vehicle; wherein the spatial gated recurrent unit is used to dynamically update the hidden state in the spatial dimension and fuse the input features and historical state information.

[0025] In a specific implementation of the first aspect, the processing of the bird's-eye view perception information and the plurality of candidate path trajectories by an environment exploration module to obtain the future driving environment of the vehicle may include:

[0026] Determining camera data and lidar data corresponding to the bird's-eye view perception information;

[0027] Performing a multi-scale deformable attention mechanism on the camera data according to the plurality of candidate path trajectories to obtain a first extracted feature;

[0028] Performing multi-layer perceptron processing on the laser radar data according to the plurality of candidate path trajectories to obtain a second extracted feature;

[0029] Performing feature splicing processing on the first extracted feature and the second extracted feature to obtain a spliced ​​feature;

[0030] The splicing features are processed by a multi-layer perceptron to obtain a future driving environment of the vehicle.

[0031] In a specific implementation of the first aspect, the processing of the future driving area and the future driving environment by a path scoring module to obtain scoring results of the multiple candidate path trajectories may include:

[0032] For the mth candidate path trajectory among the multiple candidate path trajectories, determine the scores of the sub-indicators of the mth candidate path trajectory according to the future driving area and the future driving environment; wherein 1≤m≤M, M is the number of the multiple candidate path trajectories;

[0033] According to the scores of each sub-indicator of the mth candidate path trajectory, a scoring result of the mth candidate path trajectory is determined.

[0034] In a specific implementation of the first aspect, respectively determining the scores of the sub-indicators of the m-th candidate path trajectory may include:

[0035] Determining a collision risk score, a traffic rule constraint score, and a trajectory smoothness score of the mth candidate path trajectory respectively;

[0036] Determining the scoring result of the mth candidate path trajectory according to the scores of each sub-indicator of the mth candidate path trajectory may include:

[0037] The collision risk score, the traffic rule constraint score, and the trajectory smoothness score of the mth candidate path trajectory are summed to obtain a scoring result of the mth candidate path trajectory.

[0038] A second aspect of an embodiment of the present application provides a path planning device, which may include:

[0039] A data acquisition module, used to acquire multi-view camera images and lidar point clouds of the vehicle;

[0040] a bird's-eye view perception module, configured to determine bird's-eye view perception information of the vehicle based on the multi-view camera image and the lidar point cloud;

[0041] a trajectory generation module, configured to process the bird's-eye view perception information to obtain a plurality of candidate path trajectories of the vehicle;

[0042] A driving area prediction module, used for processing the bird's-eye view perception information and the plurality of candidate path trajectories to obtain a future driving area of ​​the vehicle;

[0043] An environment exploration module, configured to process the bird's-eye view perception information and the plurality of candidate path trajectories to obtain a future driving environment of the vehicle;

[0044] A path scoring module is used to process the future driving area and the future driving environment through the path scoring module to obtain scoring results of the multiple candidate path trajectories.

[0045] In a specific implementation of the second aspect, the bird's-eye view perception module can be specifically used to: perform feature extraction on the multi-view camera image to obtain the camera image features of the vehicle; perform feature extraction on the lidar point cloud to obtain the lidar point cloud features of the vehicle; perform bird's-eye view conversion on the camera image features to obtain the camera image bird's-eye features of the vehicle; perform bird's-eye view conversion on the lidar point cloud features to obtain the lidar point cloud bird's-eye features of the vehicle; perform feature fusion on the camera image bird's-eye features and the lidar point cloud bird's-eye features to obtain the fused features of the vehicle; decode the fused features of the vehicle to obtain the bird's-eye map and target detection information of the vehicle; and determine the bird's-eye view perception information of the vehicle based on the bird's-eye map and the target detection information.

[0046] In a specific implementation of the second aspect, the trajectory generation module can be specifically used to: generate multiple candidate path heat maps based on the bird's-eye view perception information; wherein each candidate path heat map is used to characterize the probability of each grid being a path point; based on the probability of each grid being a path point, path point extraction is performed in the multiple candidate path heat maps to obtain multiple candidate path trajectories of the vehicle.

[0047] In a specific implementation of the second aspect, the driving area prediction module may include a spatial gated loop unit; the driving area prediction module may be specifically used to: iteratively process the bird's-eye view perception information and the multiple candidate path trajectories step by step in time through the spatial gated loop unit to obtain the future driving area of ​​the vehicle; wherein the spatial gated loop unit is used to dynamically update the hidden state in the spatial dimension and fuse the input features and historical state information.

[0048] In a specific implementation of the second aspect, the environment exploration module can be specifically used to: determine the camera data and lidar data corresponding to the bird's-eye view perception information; perform a multi-scale deformable attention mechanism on the camera data according to the multiple candidate path trajectories to obtain a first extracted feature; perform a multi-layer perceptron processing on the lidar data according to the multiple candidate path trajectories to obtain a second extracted feature; perform feature splicing processing on the first extracted feature and the second extracted feature to obtain a spliced ​​feature; perform a multi-layer perceptron processing on the spliced ​​feature to obtain the future driving environment of the vehicle.

[0049] In a specific implementation of the second aspect, the path scoring module can be specifically used to: for the mth candidate path trajectory among the multiple candidate path trajectories, determine the sub-indicator scores of the mth candidate path trajectory according to the future driving area and the future driving environment; wherein, 1≤m≤M, M is the number of the multiple candidate path trajectories; determine the scoring result of the mth candidate path trajectory according to the sub-indicator scores of the mth candidate path trajectory.

[0050] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned path planning methods are implemented.

[0051] A fourth aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned path planning methods when executing the computer program.

[0052] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the steps of any one of the above-mentioned path planning methods.

[0053] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain multi-view camera images and laser radar point clouds of the vehicle; determine the bird's-eye view perception information of the vehicle based on the multi-view camera images and the laser radar point clouds; process the bird's-eye view perception information through a trajectory generation module to obtain multiple candidate path trajectories of the vehicle; process the bird's-eye view perception information and the multiple candidate path trajectories through a driving area prediction module to obtain the future driving area of ​​the vehicle; process the bird's-eye view perception information and the multiple candidate path trajectories through an environment exploration module to obtain the future driving environment of the vehicle; process the future driving area and the future driving environment through a path scoring module to obtain the scoring results of the multiple candidate path trajectories. Through the embodiments of the present application, path planning can be performed without relying on high-precision maps, with low cost and wider scene applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A schematic diagram of the overall structure of the path planning network used in the embodiments of the present application;

[0056] Figure 2 This is a flow chart of an embodiment of a path planning method in an embodiment of the present application;

[0057] Figure 3 Schematic diagram of the encoder in the path planning network processing multi-view camera images and lidar point clouds;

[0058] Figure 4 A schematic diagram of the future driving area of ​​the vehicle predicted by the driving area prediction module;

[0059] Figure 5 This is a structural diagram of an embodiment of a path planning device in an embodiment of the present application;

[0060] Figure 6 This is a schematic block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0063] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0064] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0065] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0066] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0067] In the field of autonomous driving technology, existing path planning methods are highly dependent on high-precision maps. However, the construction and maintenance costs of high-precision maps are very high, and high-precision maps may not be available in special scenarios (such as underground garages, remote areas, and private areas). Therefore, a path planning method that does not rely on high-precision maps is urgently needed.

[0068] In view of this, embodiments of the present application provide a path planning method, device, computer-readable storage medium and electronic device to solve the problem of over-reliance on high-precision maps in existing path planning methods.

[0069] Figure 1 The figure shows the overall structure diagram of the end-to-end path planning network without a map (high-precision map) based on a bird's eye view (Bird's Eye View, BEV) used in the embodiment of the present application. As shown in the figure, the path planning network may include an encoder (Encoder) and a decoder (Decoder), wherein the encoder is used to perceive BEV based on multi-view camera images and laser radar point clouds, extract BEV features of multi-view camera images and laser radar point clouds respectively, and then fuse the two and decode to obtain the real-time position of the vehicle based on BEV. The decoder is used to plan the path based on BEV features, and the decoder may include but is not limited to a trajectory generation module (Transformer Decoder), a driving area prediction module (Prediction), an environment exploration module (Environment Explorer), and a path scoring module (Critic). The trajectory generation module is used to plan the path according to the bird's eye view perception information output by the encoder and the scoring result output by the path scoring module. The driving area prediction module is used to predict the future driving area of ​​the vehicle, and then the path scoring module is used to evaluate whether there is a risk of collision. Considering the real-time changing environmental information such as traffic lights, the environment exploration module can be used to extract dynamically changing environmental information and feed it back to the path scoring module. The path scoring module is used to evaluate the impact of each path on the environment (including but not limited to the impact on other vehicles) and feed it back to the trajectory generation module to optimize the path planning, thereby obtaining a better path planning result.

[0070] See also Figure 2 , an embodiment of a path planning method in the embodiment of the present application may include:

[0071] Step S201, obtaining multi-view camera images and lidar point clouds of the vehicle.

[0072] The multi-view camera image can be acquired by cameras with two or more different viewpoints in the vehicle, and the laser radar point cloud can be acquired by a laser radar (LiDAR) in the vehicle. In an embodiment of the present application, the multi-view camera image and the laser radar point cloud can be used as inputs of an encoder in a path planning network.

[0073] Step S202: Determine the bird's-eye view perception information of the vehicle based on the multi-view camera image and the lidar point cloud.

[0074] Figure 3The figure shows a schematic diagram of the encoder in the path planning network processing multi-view camera images and lidar point clouds. As shown in the figure, the encoder can extract features from the multi-view camera images and lidar point clouds respectively and convert them into BEV features, and then fuse the two to obtain fused features. By decoding the fused features and combining the fused features with the camera images and point clouds respectively, a bird's-eye view map (BEV Map) and object detection information (Object Detection) can be obtained respectively. The fusion of the two can obtain the real-time position of the vehicle from the BEV perspective.

[0075] Specifically, the multi-view camera image and the LiDAR point cloud can be first received as input, and then the multi-view camera image is subjected to feature extraction to obtain the camera image features of the vehicle; and the LiDAR point cloud is subjected to feature extraction to obtain the LiDAR point cloud features of the vehicle. Among them, the multi-view camera image can be used to extract features through the convolutional neural network ResNet, and its residual connection mechanism can effectively alleviate the gradient vanishing problem of the deep network, thereby extracting more semantic features, including environmental information such as road boundaries, lane lines, pedestrians and vehicles. The LiDAR point cloud can be used for feature extraction through VoxelNet, which first divides the point cloud into a three-dimensional voxel grid (Voxel Grid), then encodes the point cloud in the voxel through the voxel feature encoder (Voxel Feature Encoder, VFE), and uses the 3D convolutional network to extract local geometric features, and finally generates a dense three-dimensional feature map that can accurately describe the depth information and geometric shape in the environment.

[0076] After the camera image features and lidar point cloud features are extracted, they can be mapped to BEV representations respectively, that is, the camera image features are converted to a bird's-eye view to obtain the vehicle's camera image bird's-eye view features; and the lidar point cloud features are converted to a bird's-eye view to obtain the vehicle's lidar point cloud bird's-eye view features. Among them, the camera image features can be converted into a BEV feature map through geometric projection, considering the internal and external parameter information of the camera, and correctly aligning the two-dimensional features to the bird's-eye view plane; while the lidar point cloud features can generate a BEV representation containing geometric information such as density and height through dimensionality reduction and projection operations.

[0077] Subsequently, the camera image bird's-eye view features and the lidar point cloud bird's-eye view features can be fused, that is, the two BEV features are unified and integrated using weighted summation, feature concatenation or attention mechanism (dynamically adjusting feature weights according to the scene) to obtain the fused features of the vehicle. The fused BEV feature map not only combines the semantic information of the image, but also retains the precise spatial characteristics of the point cloud.

[0078] After obtaining the fused features, they can be decoded to obtain the bird's-eye view map and target detection information of the vehicle, and the bird's-eye view perception information of the vehicle can be determined based on the bird's-eye view map and target detection information. Among them, the perception of the real-time position of the vehicle is achieved through the registration of point clouds and multi-sensor data fusion. The vehicle position is corrected by combining IMU, point cloud and historical trajectory data through Kalman filtering. On the fused BEV feature map, the deep learning network (CenterPoint) can be used to detect the targets in the scene, and output accurate information including target categories (such as vehicles, pedestrians, etc.), bounding boxes (length, width, height, center point position) and directions. The bird's-eye view map is a static bird's-eye view map centered on the vehicle. The specific and accurate location information of other vehicles is obtained through the target detection algorithm. Combining the two can obtain the real-time position of the vehicle from the perspective of the BEV.

[0079] The encoder can generate a bird's-eye view map that includes the vehicle's position and objects in the environment, providing key support for downstream path planning for autonomous driving. This framework fully utilizes the complementary characteristics of visual and point cloud data by combining the semantic feature extraction capabilities of ResNet and the geometric feature capture capabilities of VoxelNet, demonstrating the efficiency and accuracy of perception in dynamic environments. This design not only retains the core advantages of multimodal features during the fusion process, but also optimizes the perception quality through geometric consistency and attention mechanisms. It is a better multimodal perception solution in the field of autonomous driving.

[0080] Step S203: Process the bird's-eye view perception information through a trajectory generation module to obtain multiple candidate path trajectories of the vehicle.

[0081] The trajectory generation module is the core part of the path planning network. It can generate multimodal candidate path trajectories and their confidences based on the environmental information provided by the encoder. In order to efficiently integrate the global information and local dynamic features from the environment, multiple candidate path heatmaps can be generated based on the Transformer Decoder structure and the bird's-eye view perception information. Each candidate path heatmap is used to characterize the probability of each grid as a waypoint, that is, the grid value in the heatmap represents the probability that the potential path point of the trajectory appears at that location. Then, the path extraction algorithm can be used to extract the path points in multiple candidate path heatmaps according to the probability of each grid as a waypoint to obtain multiple candidate path trajectories of the vehicle. At the same time, in order to further improve the accuracy of path planning, a confidence network can also be trained to dynamically evaluate the reliability of different candidate path trajectories. The core goal of the confidence network is to assign a confidence score to each candidate path trajectory to reflect its credibility in the current scene. By introducing the confidence mechanism, the contribution weight of multimodal data can be dynamically adjusted in different environments to improve the robustness of the system. The trajectory generation module not only focuses on the geometric rationality of the path trajectory, but also fully considers the environmental dynamics, such as traffic light status, obstacle distribution and traffic rules constraints, and provides multiple candidate path trajectories for subsequent modules, so that they can make reasonable choices according to specific scenarios, thereby improving the flexibility and safety of path planning.

[0082] The input of the trajectory generation module can include bird's-eye view perception information, which can include environmental features in the BEV, such as the position of vehicles and pedestrians, lane boundaries, traffic light status, and motion information of dynamic obstacles. The input of the trajectory generation module can also include the scoring results fed back by the path scoring module. These input features are fused into the Transformer Decoder in the form of tensors as context information for the trajectory generation process.

[0083] During the trajectory generation process, information from different sensors (such as cameras, lidar, etc.) is processed through a specific perceptual encoder to generate a unified multimodal feature representation. In order to capture temporal dependencies, dynamic scene information is added to the input through time series feature modeling.

[0084] The self-attention mechanism of Transformer Decoder can efficiently capture the global correlation between input features, ensuring that path planning does not only rely on local information but also considers global constraints. Through the cross-attention module, dynamic environmental features (such as traffic lights, dynamic obstacles, etc.) can be embedded in the context of trajectory generation, making the generated path trajectory dynamically adaptable.

[0085] The output of Transformer Decoder is a set of heat maps of candidate path trajectories. The heat map of each path trajectory is a two-dimensional matrix, whose value represents the probability of each grid as a path point. These heat maps are subsequently processed to extract key path points to form specific candidate path trajectories.

[0086] After generating the heat map, the trajectory generation module can extract the path points from it by using the probability maximization method. Specifically, for each candidate path trajectory, the grid value in its heat map represents the probability that the grid is a path point, as shown in the following formula:

[0087] P(w i,j )=softmax(H i,j ).

[0088] Among them, P(w i,j ) represents the path point w i,j The probability of position (i, j) in the heat map, H i,j Indicates the eigenvalue of the corresponding position in the heat map.

[0089] Dynamic programming or greedy algorithms can be used to select points with higher probability in the heat map, forming candidate path trajectories while ensuring path smoothness and physical constraints, and ultimately generating multiple diverse candidate path trajectories, each of which represents a possible driving decision.

[0090] Transformer Decoder effectively models the global correlation between environmental features through the attention mechanism, ensuring that the generated path can adapt to complex dynamic environments. By fusing the perception results of different sensors, more accurate and robust candidate path trajectories can be generated, reducing the impact of sensor failure or noise on the results. Moreover, since multiple candidate path trajectories are generated, it provides diversified options for subsequent path selection, making it easier to select the optimal path in dynamic scenes. The structure of Transformer Decoder can adapt to more feature inputs (such as new sensor data or environmental constraints in the future) and has high scalability.

[0091] Through the Transformer Decoder-based trajectory generation module, efficient path trajectory generation and deep fusion of multimodal features can be achieved, providing a reliable set of candidate paths for subsequent path selection and optimization, which is the basic guarantee for achieving safe driving in dynamic and complex environments.

[0092] The role of the confidence network is to assign confidence scores to the generated candidate path trajectories and finally select the best mode as the output. In order to improve the performance of the confidence network, an improved loss function can be designed for it to focus on the following two points: the ground truth mode (the mode closest to the true trajectory) should have the highest confidence, and the selected best mode should be consistent with the ground truth mode.

[0093] The path generation module obtains M candidate path trajectories. The confidence of each trajectory is determined by the probability value {c1, c2, …, c M} means, where c m is the confidence of the mth candidate path trajectory, satisfying:

[0094]

[0095] Let y GT is the modality number corresponding to the ground truth, y best The modal number with the highest confidence selected by the confidence network is:

[0096]

[0097] The confidence network can be constrained to assign higher confidence to candidate path trajectories corresponding to the ground truth modality through confidence distribution matching loss (Cross-Entropy):

[0098]

[0099] This design can effectively constrain the output distribution of the confidence network while improving the overall performance of the system, especially the planning quality in multimodal scenarios.

[0100] Step S204: Process the bird's-eye view perception information and the multiple candidate path trajectories through a driving area prediction module to obtain a future driving area of ​​the vehicle.

[0101] The driving area prediction module may include a spatial gated recurrent unit (Spatial-GRU), which is a variant of a GRU network that is used to process spatial sequence data. Compared with the traditional GRU, the Spatial-GRU replaces linear operations with two-dimensional convolutional layers, allowing the network to consider the spatial relationship between pixels when processing image data. This enables the Spatial-GRU to better capture features and patterns in the image, which helps improve the performance of tasks on image data. At the same time, the Spatial-GRU can well capture long-term dependencies in sequence data and can better complete prediction tasks. Figure 4 As shown in the figure, the current bird's-eye view perception information and the path trajectory of the vehicle in the next time step can be input into Spatial-GRU to obtain the drivable area in the next time step, and the driving area of ​​the vehicle in the future can be obtained by predicting it unit by unit time.

[0102] Spatial-GRU generates the next drivable area by using the bird's-eye view perception information and the vehicle state information at the current moment. In the specific implementation, the input data is first constructed as the bird's-eye view perception information containing the environment space state and the current state information of the vehicle (such as position, speed, direction, etc.). The bird's-eye view perception information can usually include lane information, obstacle positions, and dynamic object states to describe the scene around the vehicle.

[0103] After the input data is processed by Spatial-GRU, its embedded two-dimensional convolution operation replaces the linear transformation in the traditional GRU to extract the spatial features between pixels in the bird's-eye view perception information. Through the gating mechanism (including update gate and reset gate), Spatial-GRU dynamically updates the hidden state, fuses the input features and historical state information, and maintains the ability to express long-term dependencies. Unlike traditional GRU, the state update operation of Spatial-GRU is performed in the spatial dimension, and the generated hidden state directly represents the dynamic characteristics of each pixel in the scene in two dimensions.

[0104] After the state is updated, Spatial-GRU generates an output through a mapping layer, which is the drivable area for the next step. This output is usually a two-dimensional probability map, in which each pixel value represents the probability that the corresponding position is a drivable area. In an embodiment of the present application, the Spatial-GRU can be used to iteratively process the bird's-eye view perception information and multiple candidate path trajectories time step by time to obtain the future driving area of ​​the vehicle. That is, the predicted drivable area can be used as the input for the next time step, combined with the new bird's-eye view perception information and the vehicle state information, and the driving area for a period of time in the future can be iteratively predicted time step by time. In this way, Spatial-GRU can effectively capture the spatial and temporal characteristics of the input data, and provide support for trajectory prediction and path planning in dynamic environments.

[0105] Step S205: Process the bird's-eye view perception information and multiple candidate path trajectories through the environment exploration module to obtain the future driving environment of the vehicle.

[0106] The design goal of the environment exploration module is to provide more accurate spatial information support for subsequent trajectory prediction by retrieving sensor features based on predicted trajectories. This module imitates the behavior of human drivers observing the target area before taking action to avoid collisions or violating traffic rules, and is an important component in the entire system to enhance the spatial perception ability of the decoder.

[0107] In the embodiment of the present application, the camera data and lidar data corresponding to the bird's-eye view perception information can be first determined, that is, the sensor features from the camera and the lidar. For the camera data, the trajectory coordinates are projected back to the image plane using the camera's internal and external parameters, and for the lidar data, since it has been aligned to the BEV perspective, the trajectory coordinates can be used directly.

[0108] In the feature extraction stage, different methods can be used for camera data and lidar data. For camera data, a multi-scale deformable attention mechanism can be used to process the camera data according to multiple candidate path trajectories to obtain the first extracted feature, as shown in the following formula:

[0109]

[0110] Among them, H Img is a multi-scale image feature map, Traj i is the trajectory coordinate of the reference point, H env and H mst They represent the embedding vectors of environmental features and high-level navigation instructions respectively, and DeformAttn is the deformable attention mechanism processing function. Extract features for the first time.

[0111] For the lidar data, the lidar data can be processed by a multi-layer perceptron (MLP) according to multiple candidate path trajectories to obtain the second extracted features. That is, the voxel features near the trajectory coordinates are directly retrieved and the required features are obtained by multi-layer perceptron processing.

[0112] Next, the first extracted feature and the second extracted feature may be subjected to feature splicing processing to obtain a spliced ​​feature, and the spliced ​​feature may be subjected to multi-layer perceptron processing to obtain the future driving environment of the vehicle, as shown in the following formula:

[0113]

[0114] in, is the first feature extraction, is the second extracted feature, MLP is the multi-layer perceptron processing function, For the future driving environment of vehicles.

[0115] By introducing sample-specific target locations into feature representation, the environment exploration module reuses the expressive power of the encoder while improving the optimization efficiency and generalization ability of the network. This mechanism enables the network to effectively focus on potential key areas, providing important support for fine-tuning decision-making and trajectory prediction.

[0116] Step S206: Process the future driving area and the future driving environment through a path scoring module to obtain scoring results of multiple candidate path trajectories.

[0117] In the embodiment of the present application, the path scoring module is a key component for path quality assessment. Its goal is to comprehensively evaluate the candidate path trajectories, combining the interaction information between vehicles, environmental constraints (such as traffic lights and traffic rules), driving comfort, speed efficiency and other dimensions to screen out the optimal path trajectory. Specifically, the path scoring module will score each candidate path trajectory item by item and rank the trajectory quality through a comprehensive scoring function.

[0118] For the mth candidate path among the multiple candidate path trajectories, the scores of the sub-indicators of the mth candidate path trajectory can be determined respectively according to the future driving area and the future driving environment, and the score result of the mth candidate path trajectory can be determined according to the scores of the sub-indicators of the mth candidate path trajectory. Taking the example that the scores of the sub-indicators include the collision risk score, the traffic rule constraint score and the trajectory smoothness score, the collision risk score, the traffic rule constraint score and the trajectory smoothness score of the mth candidate path trajectory can be determined respectively, and the collision risk score, the traffic rule constraint score and the trajectory smoothness score of the mth candidate path trajectory can be summed to obtain the score result of the mth candidate path trajectory.

[0119] The path scoring module evaluates each candidate path trajectory. The score can be expressed as:

[0120] S m =f critic (T m ,ε),

[0121] Among them, T m is the mth candidate path trajectory, which consists of the coordinates of multiple time steps (the total number of time steps is recorded as N), is the coordinate of the mth candidate path trajectory at the tth time step, ε is the environmental information, including dynamic constraints (such as surrounding vehicle status, etc.), static constraints (such as traffic lights, obstacles, etc.) and other interactive factors, f critic is a comprehensive scoring function used to comprehensively measure the safety and rationality of the trajectory, S m is the scoring result of the mth candidate path trajectory.

[0122] Rating results m It can be divided into the sum of several sub-indicator scores:

[0123] S m =S collision (T m )+S traffic (T m ,ε)+S smoothness (T m )

[0124] S collision (T m ) is a collision risk score, which is used to measure the probability of collision between the path trajectory and other vehicles or obstacles:

[0125]

[0126] in, are the coordinates of the surrounding vehicles at the tth time step. The smaller the distance between the path trajectory of the vehicle and the path trajectory of other vehicles, the more likely the trajectory is to collide and the lower the score.

[0127] S traffic (T m ,ε) is the traffic rule constraint score, which mainly measures whether the path trajectory complies with traffic rules, such as traffic lights, stop lines or lane boundaries:

[0128]

[0129] Among them, s t is the cumulative distance traveled by the vehicle on the path trajectory, s stop is the position of the red light stop line, is the penalty points for not complying with traffic regulations at the tth time step. When the vehicle crosses the stop line, otherwise is zero.

[0130] S smoothness (T m ) is the trajectory smoothness score, which mainly measures the smoothness of the path trajectory in the time dimension to avoid overly aggressive or frequent turns:

[0131]

[0132] This formula represents the second-order difference between trajectory points, reflecting the change in curvature of the trajectory. A smooth trajectory has a smaller curvature change.

[0133] The path scoring module scores candidate trajectories and selects the best solution by comprehensively evaluating collision risk, traffic rule constraints, and trajectory smoothness. In a dynamic and complex environment, it can ensure the safety and rationality of the path trajectory and is an important part of the entire path planning framework.

[0134] In summary, the embodiment of the present application obtains the multi-view camera image and the laser radar point cloud of the vehicle; determines the bird's-eye view perception information of the vehicle based on the multi-view camera image and the laser radar point cloud; processes the bird's-eye view perception information through the trajectory generation module to obtain multiple candidate path trajectories of the vehicle; processes the bird's-eye view perception information and multiple candidate path trajectories through the driving area prediction module to obtain the future driving area of ​​the vehicle; processes the bird's-eye view perception information and multiple candidate path trajectories through the environment exploration module to obtain the future driving environment of the vehicle; processes the future driving area and the future driving environment through the path scoring module to obtain the scoring results of multiple candidate path trajectories. Through the embodiment of the present application, path planning can be performed without relying on high-precision maps, the cost is low, and it has a wider range of scene applicability.

[0135] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0136] Corresponding to a path planning method described in the above embodiment, Figure 5 A structural diagram of an embodiment of a path planning device provided in an embodiment of the present application is shown.

[0137] In this embodiment, a path planning device may include:

[0138] A data acquisition module 501 is used to acquire multi-view camera images and laser radar point clouds of the vehicle;

[0139] A bird's-eye view perception module 502, configured to determine bird's-eye view perception information of the vehicle based on the multi-view camera image and the lidar point cloud;

[0140] A trajectory generation module 503, configured to process the bird's-eye view perception information to obtain multiple candidate path trajectories of the vehicle;

[0141] A driving area prediction module 504 is used to process the bird's-eye view perception information and the plurality of candidate path trajectories to obtain a future driving area of ​​the vehicle;

[0142] An environment exploration module 505, configured to process the bird's-eye view perception information and the plurality of candidate path trajectories to obtain a future driving environment of the vehicle;

[0143] The path scoring module 506 is used to process the future driving area and the future driving environment through a path scoring module to obtain scoring results of the multiple candidate path trajectories.

[0144] In a specific implementation method of an embodiment of the present application, the bird's-eye view perception module can be specifically used to: perform feature extraction on the multi-view camera image to obtain the camera image features of the vehicle; perform feature extraction on the lidar point cloud to obtain the lidar point cloud features of the vehicle; perform bird's-eye view conversion on the camera image features to obtain the camera image bird's-eye features of the vehicle; perform bird's-eye view conversion on the lidar point cloud features to obtain the lidar point cloud bird's-eye features of the vehicle; perform feature fusion on the camera image bird's-eye features and the lidar point cloud bird's-eye features to obtain the fused features of the vehicle; decode the fused features of the vehicle to obtain the bird's-eye map and target detection information of the vehicle; and determine the bird's-eye view perception information of the vehicle based on the bird's-eye map and the target detection information.

[0145] In a specific implementation of an embodiment of the present application, the trajectory generation module can be specifically used to: generate multiple candidate path heat maps based on the bird's-eye view perception information; wherein each candidate path heat map is used to characterize the probability of each grid being a path point; based on the probability of each grid being a path point, path point extraction is performed in the multiple candidate path heat maps to obtain multiple candidate path trajectories of the vehicle.

[0146] In a specific implementation of an embodiment of the present application, the driving area prediction module may include a spatial gated loop unit; the driving area prediction module may be specifically used to: iteratively process the bird's-eye view perception information and the multiple candidate path trajectories step by step in time through the spatial gated loop unit to obtain the future driving area of ​​the vehicle; wherein the spatial gated loop unit is used to dynamically update the hidden state in the spatial dimension and fuse the input features and historical state information.

[0147] In a specific implementation method of an embodiment of the present application, the environment exploration module can be specifically used to: determine the camera data and lidar data corresponding to the bird's-eye view perception information; perform a multi-scale deformable attention mechanism on the camera data according to the multiple candidate path trajectories to obtain a first extracted feature; perform a multi-layer perceptron processing on the lidar data according to the multiple candidate path trajectories to obtain a second extracted feature; perform feature splicing processing on the first extracted feature and the second extracted feature to obtain a spliced ​​feature; perform a multi-layer perceptron processing on the spliced ​​feature to obtain the future driving environment of the vehicle.

[0148] In a specific implementation method of an embodiment of the present application, the path scoring module can be specifically used to: for the mth candidate path trajectory among the multiple candidate path trajectories, determine the scores of each sub-indicator of the mth candidate path trajectory according to the future driving area and the future driving environment; wherein, 1≤m≤M, M is the number of the multiple candidate path trajectories; determine the scoring result of the mth candidate path trajectory according to the scores of each sub-indicator of the mth candidate path trajectory.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0150] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0151] Figure 6 A schematic block diagram of an electronic device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0152] like Figure 6 As shown, the electronic device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the above-mentioned various path planning method embodiments are implemented, for example Figure 2 Alternatively, when the processor 60 executes the computer program 62, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of modules 501 to 506 are shown.

[0153] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 62 in the electronic device 6.

[0154] The electronic device 6 may include but is not limited to a vehicle-mounted computing device, a tablet computer, a desktop computer, a notebook, a PDA, and a server. Those skilled in the art will appreciate that Figure 6It is only an example of the electronic device 6 and does not constitute a limitation of the electronic device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 6 may also include input and output devices, network access devices, buses, etc.

[0155] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0156] The memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. The memory 61 may also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. Further, the memory 61 may also include both an internal storage unit of the electronic device 6 and an external storage device. The memory 61 is used to store the computer program and other programs and data required by the electronic device 6. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0157] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0158] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0159] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0160] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0161] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0163] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0164] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A path planning method, characterized in that: include: Acquire multi-view camera images and lidar point clouds of the vehicle; Determining bird's-eye view perception information of the vehicle based on the multi-view camera image and the lidar point cloud; Processing the bird's-eye view perception information through a trajectory generation module to obtain multiple candidate path trajectories of the vehicle; Processing the bird's-eye view perception information and the plurality of candidate path trajectories through a driving area prediction module to obtain a future driving area of ​​the vehicle; Processing the bird's-eye view perception information and the plurality of candidate path trajectories through an environment exploration module to obtain a future driving environment of the vehicle; The future driving area and the future driving environment are processed by a path scoring module to obtain scoring results of the multiple candidate path trajectories.

2. The path planning method according to claim 1, characterized in that: The determining, according to the multi-view camera image and the laser radar point cloud, the bird's-eye view perception information of the vehicle comprises: Performing feature extraction on the multi-view camera image to obtain camera image features of the vehicle; Performing feature extraction on the laser radar point cloud to obtain laser radar point cloud features of the vehicle; Performing a bird's-eye view conversion on the camera image features to obtain a camera image bird's-eye view feature of the vehicle; Performing a bird's-eye view conversion on the laser radar point cloud features to obtain a bird's-eye view feature of the laser radar point cloud of the vehicle; Performing feature fusion on the bird's-eye view features of the camera image and the bird's-eye view features of the laser radar point cloud to obtain fused features of the vehicle; Decoding the fusion features of the vehicle to obtain a bird's-eye view map and target detection information of the vehicle; The bird's-eye view perception information of the vehicle is determined based on the bird's-eye view map and the target detection information.

3. The path planning method according to claim 1, characterized in that: The process of processing the bird's-eye view perception information by a trajectory generation module to obtain a plurality of candidate path trajectories of the vehicle includes: Generating a plurality of candidate path heat maps according to the bird's-eye view perception information; wherein each candidate path heat map is used to characterize the probability of each grid being a path point; According to the probability of each grid being a path point, path points are extracted in the multiple candidate path heat maps to obtain multiple candidate path trajectories of the vehicle.

4. The path planning method according to claim 1, characterized in that: The driving area prediction module includes a spatial gated recurrent unit; The step of processing the bird's-eye view perception information and the plurality of candidate path trajectories through a driving area prediction module to obtain a future driving area of ​​the vehicle includes: The spatial gated recurrent unit is used to iteratively process the bird's-eye view perception information and the multiple candidate path trajectories step by step in time to obtain the future driving area of ​​the vehicle; wherein the spatial gated recurrent unit is used to dynamically update the hidden state in the spatial dimension and fuse the input features and historical state information.

5. The path planning method according to claim 1, characterized in that: The step of processing the bird's-eye view perception information and the plurality of candidate path trajectories through an environment exploration module to obtain a future driving environment of the vehicle includes: Determining camera data and lidar data corresponding to the bird's-eye view perception information; Performing a multi-scale deformable attention mechanism on the camera data according to the plurality of candidate path trajectories to obtain a first extracted feature; Performing multi-layer perceptron processing on the laser radar data according to the plurality of candidate path trajectories to obtain a second extracted feature; Performing feature splicing processing on the first extracted feature and the second extracted feature to obtain a spliced ​​feature; The splicing features are processed by a multi-layer perceptron to obtain a future driving environment of the vehicle.

6. The path planning method according to any one of claims 1 to 5, characterized in that: The future driving area and the future driving environment are processed by the path scoring module to obtain scoring results of the multiple candidate path trajectories, including: For the mth candidate path trajectory among the multiple candidate path trajectories, determine the scores of the sub-indicators of the mth candidate path trajectory according to the future driving area and the future driving environment; wherein 1≤m≤M, M is the number of the multiple candidate path trajectories; According to the scores of each sub-indicator of the mth candidate path trajectory, a scoring result of the mth candidate path trajectory is determined.

7. The path planning method according to claim 6, characterized in that: The step of respectively determining the scores of the sub-indicators of the m-th candidate path trajectory comprises: Determining a collision risk score, a traffic rule constraint score, and a trajectory smoothness score of the mth candidate path trajectory respectively; The step of determining the scoring result of the mth candidate path trajectory according to the scores of each sub-indicator of the mth candidate path trajectory comprises: The collision risk score, the traffic rule constraint score, and the trajectory smoothness score of the mth candidate path trajectory are summed to obtain a scoring result of the mth candidate path trajectory.

8. A path planning device, characterized in that: include: A data acquisition module, used to acquire multi-view camera images and lidar point clouds of the vehicle; a bird's-eye view perception module, configured to determine bird's-eye view perception information of the vehicle based on the multi-view camera image and the lidar point cloud; a trajectory generation module, configured to process the bird's-eye view perception information to obtain a plurality of candidate path trajectories of the vehicle; A driving area prediction module, used for processing the bird's-eye view perception information and the plurality of candidate path trajectories to obtain a future driving area of ​​the vehicle; An environment exploration module, configured to process the bird's-eye view perception information and the plurality of candidate path trajectories to obtain a future driving environment of the vehicle; A path scoring module is used to process the future driving area and the future driving environment through the path scoring module to obtain scoring results of the multiple candidate path trajectories.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the path planning method according to any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the path planning method according to any one of claims 1 to 7 are implemented.

Images