Path planning method and device, computer readable storage medium and electronic device

Abstract

The application belongs to the technical field of automatic driving, and particularly relates to a path planning method and device, a computer readable storage medium and an electronic device. The method comprises the following steps: acquiring a multi-view camera image and a laser radar point cloud of a vehicle; determining bird's-eye view perception information of the vehicle according to the multi-view camera image and the laser radar point cloud; processing the bird's-eye view perception information through a trajectory generation module to obtain a plurality of candidate path trajectories of the vehicle; processing the bird's-eye view perception information and the plurality of candidate path trajectories through a driving area prediction module to obtain a future driving area of the vehicle; processing the bird's-eye view perception information and the plurality of candidate path trajectories through an environment exploration module to obtain a future driving environment of the vehicle; and processing the future driving area and the future driving environment through a path scoring module to obtain a scoring result of the plurality of candidate path trajectories.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automatic driving, and particularly relates to a path planning method and device, a computer readable storage medium, and an electronic device. BACKGROUND

[0002] In the technical field of automatic driving, the existing path planning method is highly dependent on high-precision maps, but the construction and maintenance of high-precision maps are very costly, and high-precision maps may not be available in special scenarios (such as underground garages, remote areas, and private areas). Therefore, there is an urgent need for a path planning method that does not depend on high-precision maps. SUMMARY

[0003] Therefore, the embodiments of the present application provide a path planning method, device, computer readable storage medium, and electronic device to solve the problem that the existing path planning method is too dependent on high-precision maps.

[0004] A first aspect of the embodiments of the present application provides a path planning method, which can include:

[0005] Obtaining a multi-view camera image and a laser radar point cloud of a vehicle;

[0006] Determining bird's-eye view perception information of the vehicle according to the multi-view camera image and the laser radar point cloud;

[0007] Processing the bird's-eye view perception information through a trajectory generation module to obtain a plurality of candidate path trajectories of the vehicle;

[0008] Processing the bird's-eye view perception information and the plurality of candidate path trajectories through a driving area prediction module to obtain a future driving area of the vehicle;

[0009] Processing the bird's-eye view perception information and the plurality of candidate path trajectories through an environment exploration module to obtain a future driving environment of the vehicle;

[0010] Processing the future driving area and the future driving environment through a path scoring module to obtain a scoring result of the plurality of candidate path trajectories.

[0011] In a specific implementation manner of the first aspect, the determining the bird's-eye view perception information of the vehicle according to the multi-view camera image and the laser radar point cloud can include:

[0012] Performing feature extraction on the multi-view camera image to obtain camera image features of the vehicle;

[0013] Performing feature extraction on the laser radar point cloud to obtain laser radar point cloud features of the vehicle;

[0014] performing bird's eye view conversion on the camera image features to obtain camera image bird's eye view features of the vehicle;

[0015] performing bird's eye view conversion on the lidar point cloud features to obtain lidar point cloud bird's eye view features of the vehicle;

[0016] performing feature fusion on the camera image bird's eye view features and the lidar point cloud bird's eye view features to obtain fusion features of the vehicle;

[0017] decoding the fusion features of the vehicle to obtain a bird's eye view map and target detection information of the vehicle;

[0018] determining bird's eye view perception information of the vehicle according to the bird's eye view map and the target detection information.

[0019] In an implementation form of the first aspect, the processing, by the trajectory generation module, of the bird's eye view perception information to obtain a plurality of candidate path trajectories of the vehicle can include:

[0020] generating a plurality of candidate path heat maps according to the bird's eye view perception information; wherein each candidate path heat map is used to represent a probability of each grid as a path point;

[0021] extracting path points in the plurality of candidate path heat maps according to the probability of each grid as a path point to obtain the plurality of candidate path trajectories of the vehicle.

[0022] In an implementation form of the first aspect, the driving area prediction module includes a spatial gating recurrent unit;

[0023] The processing, by the driving area prediction module, of the bird's eye view perception information and the plurality of candidate path trajectories to obtain a future driving area of the vehicle can include:

[0024] iteratively processing, by the spatial gating recurrent unit, the bird's eye view perception information and the plurality of candidate path trajectories step by step in time to obtain the future driving area of the vehicle; wherein the spatial gating recurrent unit is used to dynamically update a hidden state in a spatial dimension, fuse input features and historical state information.

[0025] In an implementation form of the first aspect, the processing, by the environment exploration module, of the bird's eye view perception information and the plurality of candidate path trajectories to obtain a future driving environment of the vehicle can include:

[0026] determining camera data and lidar data corresponding to the bird's eye view perception information;

[0027] perform multi-scale deformable attention mechanism processing on the camera data according to the plurality of candidate path trajectories, to obtain first extracted features;

[0028] perform multi-layer perception processing on the lidar data according to the plurality of candidate path trajectories, to obtain second extracted features;

[0029] perform feature splicing processing on the first extracted features and the second extracted features, to obtain spliced features;

[0030] perform multi-layer perception processing on the spliced features, to obtain a future driving environment of the vehicle.

[0031] In an implementation form of the first aspect, the processing, by the path scoring module, of the future driving area and the future driving environment to obtain the score results of the plurality of candidate path trajectories can include:

[0032] for an m th candidate path trajectory in the plurality of candidate path trajectories, determining respective sub-index scores of the m th candidate path trajectory according to the future driving area and the future driving environment, where 1≤m≤M, and M is the number of the plurality of candidate path trajectories;

[0033] determining a score result of the m th candidate path trajectory according to the respective sub-index scores of the m th candidate path trajectory.

[0034] In an implementation form of the first aspect, the determining of the respective sub-index scores of the m th candidate path trajectory can include:

[0035] respectively determining a collision risk score, a traffic rule constraint score, and a trajectory smoothness score of the m th candidate path trajectory;

[0036] The determining of the score result of the m th candidate path trajectory according to the respective sub-index scores of the m th candidate path trajectory can include:

[0037] performing summation on the collision risk score, the traffic rule constraint score, and the trajectory smoothness score of the m th candidate path trajectory, to obtain the score result of the m th candidate path trajectory.

[0038] A second aspect of the embodiments of the present application provides a path planning device, which can include:

[0039] a data acquisition module configured to acquire multi-view camera images and lidar point clouds of a vehicle;

[0040] an aerial view perception module configured to determine aerial view perception information of the vehicle according to the multi-view camera images and the lidar point cloud;

[0041] a trajectory generation module configured to process the aerial view perception information to obtain a plurality of candidate path trajectories of the vehicle;

[0042] a driving area prediction module configured to process the aerial view perception information and the plurality of candidate path trajectories to obtain a future driving area of the vehicle;

[0043] an environment exploration module configured to process the aerial view perception information and the plurality of candidate path trajectories to obtain a future driving environment of the vehicle;

[0044] a path scoring module configured to process the future driving area and the future driving environment by the path scoring module to obtain a scoring result of the plurality of candidate path trajectories.

[0045] In an implementation form of the second aspect, the aerial view perception module can be specifically configured to: perform feature extraction on the multi-view camera images to obtain camera image features of the vehicle; perform feature extraction on the lidar point cloud to obtain lidar point cloud features of the vehicle; perform aerial view conversion on the camera image features to obtain camera image aerial view features of the vehicle; perform aerial view conversion on the lidar point cloud features to obtain lidar point cloud aerial view features of the vehicle; perform feature fusion on the camera image aerial view features and the lidar point cloud aerial view features to obtain fusion features of the vehicle; decode the fusion features of the vehicle to obtain an aerial view map and target detection information of the vehicle; and determine the aerial view perception information of the vehicle according to the aerial view map and the target detection information.

[0046] In an implementation form of the second aspect, the trajectory generation module can be specifically configured to: generate a plurality of candidate path heat maps according to the aerial view perception information; wherein each candidate path heat map is used to represent a probability of each grid as a path point; and perform path point extraction in the plurality of candidate path heat maps according to the probability of each grid as a path point to obtain the plurality of candidate path trajectories of the vehicle.

[0047] In an implementation form of the second aspect, the driving area prediction module can include a spatial gated recurrent unit; the driving area prediction module can be specifically configured to: perform time step by time step iteration processing on the bird's eye view perception information and the plurality of candidate path trajectories by the spatial gated recurrent unit to obtain the future driving area of the vehicle; and the spatial gated recurrent unit is configured to dynamically update a hidden state in a spatial dimension, and fuse input features and historical state information.

[0048] In an implementation form of the second aspect, the environment exploration module can be specifically configured to: determine camera data and lidar data corresponding to the bird's eye view perception information; perform multi-scale deformable attention mechanism processing on the camera data according to the plurality of candidate path trajectories to obtain first extracted features; perform multi-layer perception processing on the lidar data according to the plurality of candidate path trajectories to obtain second extracted features; perform feature splicing processing on the first extracted features and the second extracted features to obtain spliced features; and perform multi-layer perception processing on the spliced features to obtain the future driving environment of the vehicle.

[0049] In an implementation form of the second aspect, the path scoring module can be specifically configured to: for an m th candidate path trajectory in the plurality of candidate path trajectories, determine respective sub-index scores of the m th candidate path trajectory according to the future driving area and the future driving environment, where 1≤m≤M, and M is the number of the plurality of candidate path trajectories; and determine a score result of the m th candidate path trajectory according to the respective sub-index scores of the m th candidate path trajectory.

[0050] The third aspect of the embodiments of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of any of the path planning methods.

[0051] The fourth aspect of the embodiments of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of any of the path planning methods when executing the computer program.

[0052] The fifth aspect of the embodiments of the present application provides a computer program product, which, when executed on an electronic device, causes the electronic device to perform the steps of any of the path planning methods.

[0053] The beneficial effects of the embodiments of the present application compared with the prior art are: the embodiments of the present application obtain a multi-view camera image and a laser radar point cloud of a vehicle; determine bird's eye view perception information of the vehicle according to the multi-view camera image and the laser radar point cloud; process the bird's eye view perception information through a trajectory generation module to obtain a plurality of candidate path trajectories of the vehicle; process the bird's eye view perception information and the plurality of candidate path trajectories through a driving area prediction module to obtain a future driving area of the vehicle; process the bird's eye view perception information and the plurality of candidate path trajectories through an environment exploration module to obtain a future driving environment of the vehicle; and process the future driving area and the future driving environment through a path scoring module to obtain a scoring result of the plurality of candidate path trajectories. Through the embodiments of the present application, path planning can be performed without relying on a high-precision map, the cost is low, and the embodiments have wider scene applicability. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0055] Figure 1 The general structure diagram of the path planning network used in the embodiments of the present application;

[0056] Figure 2 The flowchart of one embodiment of a path planning method in the embodiments of the present application;

[0057] Figure 3 The schematic diagram of processing the multi-view camera image and the laser radar point cloud by the encoder in the path planning network;

[0058] Figure 4 The schematic diagram of predicting the future driving area of the vehicle by the driving area prediction module;

[0059] Figure 5 The structure diagram of one embodiment of a path planning device in the embodiments of the present application;

[0060] Figure 6 The schematic block diagram of an electronic device in the embodiments of the present application. DETAILED DESCRIPTION

[0061] In order to make the purposes, characteristics and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the embodiments described below are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0062] It should be understood that when used in the specification and the appended claims, the term "comprise" indicates the presence of a described feature, integer, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0063] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0064] It should be further understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0065] As used in the present application specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to a determination" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted as meaning "upon determining" or "in response to determining" or "upon detecting [the described condition or event]" or "in response to detecting [the described condition or event]" depending on the context.

[0066] In addition, in the description of the present application, the terms "first", "second", "third" and the like are only used for differentiation in description, and cannot be understood as indicating or implying relative importance.

[0067] In the field of automatic driving technology, the existing path planning method has a high dependence on high-precision maps, but the construction and maintenance cost of high-precision maps is high, and high-precision maps cannot be obtained in special scenarios (such as underground garages, remote areas, and private areas, etc.), so there is an urgent need for a path planning method that does not depend on high-precision maps.

[0068] Therefore, an embodiment of the present application provides a path planning method and device, a computer readable storage medium and an electronic device to solve the problem that the existing path planning method is too dependent on high-definition maps.

[0069] Figure 1 The figure shows the overall structure of the no-map (high-definition map) end-to-end path planning network based on Bird's Eye View (BEV) used in the embodiment of the present application. As shown in the figure, the path planning network can include an encoder and a decoder, wherein the encoder is used to perform BEV perception based on multi-view camera images and LiDAR point clouds, extract BEV features of the multi-view camera images and LiDAR point clouds respectively, and then decode the fusion of the two to obtain the BEV-based real-time position of the vehicle. The decoder is used to plan the path based on the BEV features. The decoder can include but is not limited to a trajectory generation module (Transformer Decoder), a driving area prediction module (Prediction), an environment exploration module (Environment Explorer), and a path scoring module (Critic), etc. The trajectory generation module is used to plan the path according to the Bird's Eye View perception information output by the encoder and the scoring results output by the path scoring module. The driving area prediction module is used to predict the future driving area of the vehicle, and then the path scoring module is used to evaluate whether there is a risk of collision, etc. Considering the real-time changing environmental information such as traffic lights, the environment exploration module can be used to extract dynamic changing environmental information and feed it back to the path scoring module. The path scoring module is used to evaluate the impact (which can include but is not limited to the impact on other vehicles) of each path on the environment and feed it back to the trajectory generation module to optimize the path planning, so as to obtain a better path planning result.

[0070] Please refer to Figure 2 An embodiment of a path planning method in the embodiment of the present application can include:

[0071] Step S201, acquiring multi-view camera images and LiDAR point clouds of a vehicle.

[0072] The multi-view camera images can be collected by two or more cameras with different perspectives in the vehicle, and the LiDAR point clouds can be collected by a LiDAR in the vehicle. In the embodiment of the present application, the multi-view camera images and the LiDAR point clouds can be used as the input of the encoder in the path planning network.

[0073] Step S202, determining the Bird's Eye View perception information of the vehicle according to the multi-view camera images and the LiDAR point clouds.

[0074] Figure 3An encoder in the path planning network processes multi-view camera images and lidar point clouds is shown in the schematic diagram. As shown in the figure, the encoder can extract features from the multi-view camera images and the lidar point clouds respectively and convert them into BEV features, and then fuse the two to obtain fused features. By decoding the fused features, the fused features are combined with the camera images and the point clouds respectively, and the bird's eye view map (BEV Map) and the object detection information (Object Detection) can be obtained respectively, and the real-time position of the vehicle in the BEV view can be obtained by fusing the two.

[0075] Specifically, the multi-view camera images and the lidar point clouds can be first received as inputs, and then the multi-view camera images are subjected to feature extraction to obtain camera image features of the vehicle, and the lidar point clouds are subjected to feature extraction to obtain lidar point cloud features of the vehicle. The multi-view camera images can be subjected to feature extraction by using a convolutional neural network ResNet, and the residual connection mechanism thereof can effectively alleviate the gradient disappearance problem of a deep network, thereby extracting features with a higher semantic level, including environmental information such as road boundaries, lane lines, pedestrians, and vehicles. The lidar point clouds can be subjected to feature extraction by using VoxelNet, which first divides the point clouds into a three-dimensional voxel grid, then encodes the point clouds in the voxels by using a voxel feature encoder (VFE), and extracts local geometric features by using a 3D convolutional network, and finally generates a dense three-dimensional feature map that can accurately describe the depth information and geometric shapes in the environment.

[0076] After the camera image features and the lidar point cloud features are extracted, the two can be mapped into BEV representations, i.e., the camera image features are subjected to bird's eye view conversion to obtain camera image bird's eye view features of the vehicle, and the lidar point cloud features are subjected to bird's eye view conversion to obtain lidar point cloud bird's eye view features of the vehicle. The camera image features can be converted into a BEV feature map by geometric projection, considering the internal and external parameter information of the camera, to correctly align the two-dimensional features to the bird's eye plane; and the lidar point cloud features can be converted into a BEV representation containing geometric information such as density and height by dimension reduction and projection operations.

[0077] Subsequently, the camera image bird's eye view features and the lidar point cloud bird's eye view features can be subjected to feature fusion, i.e., the two BEV features are uniformly integrated by using weighted summation, feature splicing, or an attention mechanism (adjusting feature weights according to the scene), thereby obtaining fused features of the vehicle. The fused BEV feature map not only combines the semantic information of the images, but also retains the accurate spatial characteristics of the point clouds.

[0078] After obtaining the fused features, the fused features can be decoded to obtain a bird's eye view map and target detection information of the vehicle, and the bird's eye view perception information of the vehicle is determined according to the bird's eye view map and the target detection information. Wherein, the perception of the real-time position of the vehicle is realized through registration of point clouds and multi-sensor data fusion. The position of the vehicle is corrected through Kalman filtering combined with IMU, point clouds and historical trajectory data. On the fused BEV feature map, a deep learning network (CenterPoint) can be used to detect targets in the scene, and accurate information including target categories (such as vehicles, pedestrians, etc.), bounding boxes (length, width, height, center point position) and directions are output. The bird's eye view map is a bird's eye view static map centered on the ego vehicle, and the specific and accurate position information of other vehicles is obtained through the target detection algorithm, and the real-time position of the vehicle in the BEV view can be obtained by combining the two.

[0079] The encoder can generate a bird's eye view map containing the position of the ego vehicle and targets in the environment, which provides key support for downstream path planning of autonomous driving. Through the combination of the semantic feature extraction ability of ResNet and the geometric feature capture ability of VoxelNet, the framework fully utilizes the complementary characteristics of visual and point cloud data, and exhibits high efficiency and accuracy in perception in dynamic environments. The design not only retains the core advantages of multi-modal features in the fusion process, but also optimizes the perception quality through geometric consistency and attention mechanism, which is a better multi-modal perception solution in the field of autonomous driving.

[0080] In step S203, the bird's eye view perception information is processed by the trajectory generation module to obtain a plurality of candidate path trajectories of the vehicle.

[0081] The trajectory generation module is the core part of the path planning network, which can generate multi-modal candidate path trajectories and their confidence scores based on the environment information provided by the encoder. In order to efficiently fuse global information from the environment and local dynamic features, multiple candidate path heatmaps can be generated based on the Transformer Decoder structure according to the bird's eye view perception information. Each candidate path heatmap is used to represent the probability of each grid as a waypoint, i.e. the grid value in the heatmap represents the probability of the potential path point of the trajectory appearing at that position. Then, through a path extraction algorithm, path points can be extracted from multiple candidate path heatmaps according to the probability of each grid as a path point, and multiple candidate path trajectories of the vehicle can be obtained. In order to further improve the accuracy of path planning, a confidence network can also be trained to dynamically evaluate the reliability of different candidate path trajectories. The core goal of the confidence network is to assign a confidence score to each candidate path trajectory, reflecting its credibility in the current scene. By introducing the confidence mechanism, the contribution weight of multi-modal data can be dynamically adjusted in different environments, improving the robustness of the system. The trajectory generation module not only focuses on the geometric rationality of the path trajectory, but also fully considers the environmental dynamics, such as traffic light status, obstacle distribution and traffic rule constraints, providing multiple candidate path trajectories for the subsequent modules, so that reasonable selection can be made according to the specific scene, improving the flexibility and safety of path planning.

[0082] The input of the trajectory generation module can include bird's eye view perception information, which can include environmental features in the BEV, such as the positions of vehicles and pedestrians, lane boundaries, traffic light status, and dynamic obstacle motion information, etc. The input of the trajectory generation module can also include the score results fed back by the path scoring module. These input features are fused into the Transformer Decoder in the form of tensors as context information for the trajectory generation process.

[0083] In the process of trajectory generation, information from different sensors (such as cameras, lidar, etc.) is processed by a specific perception encoder to generate a unified multi-modal feature representation. In order to capture temporal dependencies, dynamic scene information is modeled through time series features and added to the input.

[0084] The self-attention mechanism of the Transformer Decoder can efficiently capture the global correlation between input features, ensuring that path planning not only relies on local information, but also considers global constraints. Through the cross-attention module, dynamic environmental features (such as traffic lights, dynamic obstacles, etc.) can be embedded into the context of trajectory generation, making the generated path trajectory have dynamic adaptability.

[0085] The output of the Transformer Decoder is a set of heatmaps of candidate path trajectories, and each heatmap of a path trajectory is a two-dimensional matrix whose values represent the probability of each grid as a path point. These heatmaps are processed to extract key path points and form specific candidate path trajectories.

[0086] After generating the heatmaps, the trajectory generation module can extract path points from them using a probability maximization method. Specifically, for each candidate path trajectory, the grid values in its heatmap represent the probability of the grid as a path point, as shown in the following formula:

[0087] P(w i,j )=softmax(H i,j ).

[0088] where P(w i,j ) represents the probability of path point w M in position (i,j) in the heatmap, and H m represents the feature value at the corresponding position of the heatmap.

[0089] By using dynamic programming or a greedy algorithm, we can select points with higher probabilities in the heatmap to ensure path smoothness and physical constraints while forming candidate path trajectories. Ultimately, we generate multiple diverse candidate path trajectories, each representing a possible driving decision.

[0090] The Transformer Decoder effectively models the global relationships between environmental features through attention mechanisms, ensuring that the generated paths can adapt to complex and dynamic environments. By fusing the perception results of different sensors, we can generate more accurate and robust candidate path trajectories, reducing the impact of sensor failures or noise on the results. Moreover, the generation of multiple candidate path trajectories provides diverse options for subsequent path selection, facilitating the selection of the optimal path in dynamic scenarios. The structure of the Transformer Decoder can be adapted to more feature inputs (such as future new sensor data or environmental constraints), making it highly expandable.

[0091] Through the trajectory generation module based on the Transformer Decoder, we can achieve efficient path trajectory generation and deep fusion of multi-modal features, providing a reliable set of candidate paths for subsequent path selection and optimization. This is a fundamental guarantee for safe driving in dynamic and complex environments.

[0092] The role of the confidence network is to assign confidence scores to the generated candidate path trajectories and ultimately select the optimal mode as the output. To improve the performance of the confidence network, an improved loss function can be designed to focus on the following two points: the ground truth mode (the mode closest to the true trajectory) should have the highest confidence, and the selected optimal mode should be consistent with the ground truth mode.

[0093] The trajectory generation module obtains M candidate path trajectories. The confidence score of each trajectory is the probability value {c1,c2,…,c...} output by the confidence network. M} indicates that, where c m Let the confidence level of the m-th candidate path trajectory satisfy:

[0094]

[0095] Let y GT y is the modality number corresponding to the ground truth. best The mode number selected by the confidence network with the highest confidence level is:

[0096]

[0097] The confidence network can be constrained to assign higher confidence to candidate path trajectories corresponding to the ground truth modality by using the confidence distribution matching loss (Cross-Entropy).

[0098]

[0099] This design can effectively constrain the output distribution of the confidence network while improving the overall system performance, especially the planning quality in multimodal scenarios.

[0100] Step S204: The bird's-eye view perception information and multiple candidate path trajectories are processed by the driving area prediction module to obtain the future driving area of ​​the vehicle.

[0101] The driving region prediction module may include a Spatial-Gated Recurrent Unit (SGRU), a variant of the GRU network used to process spatial sequence data. Compared to traditional GRUs, Spatial-GRU replaces linear operations with two-dimensional convolutional layers, allowing the network to consider spatial relationships between pixels when processing image data. This enables Spatial-GRU to better capture features and patterns in images, contributing to improved performance on image data. Furthermore, Spatial-GRU effectively captures long-term dependencies in sequence data, resulting in better prediction performance.Figure 4 As shown, the current bird's eye view perception information and the path trajectory of the ego vehicle at the next time step can be input into the Spatial-GRU to obtain the drivable area at the next time step, and the driving area of the vehicle in the future period of time can be obtained by prediction per unit time.

[0102] The Spatial-GRU generates the drivable area at the next step by using the bird's eye view perception information at the current time and the ego vehicle state information. In a specific implementation, first, the input data is constructed to include the bird's eye view perception information of the environment space state and the current state information (such as position, speed, direction, etc.) of the ego vehicle. The bird's eye view perception information can generally include lane information, obstacle position, and dynamic object state, for describing the scene around the vehicle.

[0103] After the input data is processed by the Spatial-GRU, a two-dimensional convolution operation embedded in the Spatial-GRU is used to replace the linear transformation in the traditional GRU, so as to extract the spatial features between pixels in the bird's eye view perception information. Through the gating mechanism (including the update gate and the reset gate), the Spatial-GRU dynamically updates the hidden state, fuses the input features and the historical state information, and maintains the expression ability of long-time dependence. Unlike the traditional GRU, the state update operation of the Spatial-GRU is performed in the spatial dimension, and the generated hidden state directly represents the dynamic characteristics of each pixel in the scene in a two-dimensional form.

[0104] After the state is updated, the Spatial-GRU generates an output through a mapping layer, that is, the drivable area at the next step. The output is usually a two-dimensional probability map, in which each pixel value represents the probability that the corresponding position is a drivable area. In the embodiments of the present application, the bird's eye view perception information and the multiple candidate path trajectories can be processed by the Spatial-GRU in a time step by time step iteration, so as to obtain the future driving area of the vehicle. That is, the predicted drivable area can be taken as the input at the next time step, and the new bird's eye view perception information and the ego vehicle state information are combined to iteratively predict the driving area in the future period of time. In this way, the Spatial-GRU can effectively capture the spatial and temporal characteristics of the input data, and provide support for trajectory prediction and path planning in a dynamic environment.

[0105] In step S205, the bird's eye view perception information and the multiple candidate path trajectories are processed by the environment exploration module to obtain the future driving environment of the vehicle.

[0106] The design goal of the environment exploration module is to provide more accurate spatial information support for subsequent trajectory prediction by retrieving sensor features based on predicted trajectories. This module simulates the behavior of human drivers observing the target area before taking action to avoid collisions or traffic violations, and is an important component of the entire system to enhance the spatial perception ability of the decoder.

[0107] In the embodiments of the present application, camera data and lidar data corresponding to the bird's eye view perception information can be determined first, that is, sensor features from the camera and the lidar. For camera data, trajectory coordinates are projected back to the image plane using the internal and external parameters of the camera, while for lidar data, trajectory coordinates can be used directly since they have been aligned to the BEV view.

[0108] In the feature extraction stage, different methods can be used for camera data and lidar data respectively. For camera data, multi-scale deformable attention mechanism processing can be performed on the camera data according to multiple candidate path trajectories to obtain first extracted features, as shown in the following formula:

[0109]

[0110] wherein H Img is a multi-scale image feature map, Traj i is a trajectory coordinate as a reference point, H env and H mst represent embedding vectors of environmental features and high-level navigation instructions respectively, DeformAttn is a deformable attention mechanism processing function, is the first extracted feature.

[0111] For lidar data, multi-layer perceptron (MLP) processing can be performed on the lidar data according to multiple candidate path trajectories to obtain second extracted features. That is, by directly retrieving voxel features near the trajectory coordinates, and through multi-layer perceptron processing, the required features are obtained.

[0112] Then, the first extracted features and the second extracted features can be subjected to feature splicing processing to obtain spliced features, and the spliced features can be subjected to multi-layer perceptron processing to obtain the future driving environment of the vehicle, as shown in the following formula:

[0113]

[0114] wherein, is the first extracted feature, is the second extracted feature, MLP is a multi-layer perceptron processing function, is the future driving environment of the vehicle.

[0115] By introducing the sample-specific target position into the feature representation, the environment exploration module reuses the expressive power of the encoder while improving the optimization efficiency and generalization ability of the network. This mechanism enables the network to effectively focus on the potential key areas, providing important support for the refinement of decision-making and trajectory prediction.

[0116] In step S206, the future driving area and the future driving environment are processed by the path scoring module to obtain scoring results of the plurality of candidate path trajectories.

[0117] In the embodiments of the present application, the path scoring module is a key component for path quality evaluation. Its goal is to comprehensively evaluate the candidate path trajectories, combining the interaction information between vehicles, environmental constraints (such as traffic lights and traffic rules), driving comfort, and speed efficiency, etc. multiple dimensions to filter out the optimal path trajectory. Specifically, the path scoring module will score each candidate path trajectory item by item, and sort the trajectory quality through a comprehensive scoring function.

[0118] For the mth candidate path trajectory in the plurality of candidate path trajectories, each sub-index score of the mth candidate path trajectory can be determined according to the future driving area and the future driving environment, and the scoring result of the mth candidate path trajectory can be determined according to each sub-index score of the mth candidate path trajectory. Taking each sub-index score including the collision risk score, the traffic rule constraint score and the trajectory smoothness score as an example, the collision risk score, the traffic rule constraint score and the trajectory smoothness score of the mth candidate path trajectory can be determined respectively, and the collision risk score, the traffic rule constraint score and the trajectory smoothness score of the mth candidate path trajectory are summed up, thereby obtaining the scoring result of the mth candidate path trajectory.

[0119] The scoring of each candidate path trajectory by the path scoring module can be represented as:

[0120] S m =f critic (T m ,ε),

[0121] where T m is the mth candidate path trajectory composed of coordinates of multiple time steps (the total number of time steps is N), T critic is the coordinate of the mth candidate path trajectory at the tth time step, ε is the environmental information including dynamic constraints (such as the state of surrounding vehicles), static constraints (such as traffic lights, obstacles, etc.) and other interaction factors, f m is a comprehensive scoring function for comprehensively measuring the safety and rationality of the trajectory, and S m is the scoring result of the mth candidate path trajectory. ​

[0122] Score result S m The score can be divided into the sum of several sub-index scores:

[0123] S m = S collision (T m ) + S traffic (T m , ε) + S smoothness (T m )

[0124] S collision (T m ) is the collision risk score, which measures the likelihood of collision between the path trajectory and other vehicles or obstacles:

[0125]

[0126] wherein, is the coordinate of the surrounding vehicle at the t-th time step, the smaller the distance between the path trajectory of the ego vehicle and the path trajectory of other vehicles, the more likely the collision occurs, and the lower the score.

[0127] S traffic (T m , ε) is the traffic rule constraint score, which mainly measures whether the path trajectory complies with traffic rules such as traffic lights, stop lines or lane boundaries:

[0128]

[0129] wherein, s t is the cumulative driving distance of the vehicle on the path trajectory, s stop is the position of the stop line of the red light, is the deduction score at the t-th time step due to non-compliance with traffic rules, when the vehicle crosses the stop line, otherwise is zero.

[0130] S smoothness (T m ) is the trajectory smoothness score, which mainly measures the smoothness of the path trajectory in the time dimension, avoiding too aggressive or frequent steering:

[0131]

[0132] The formula represents the second-order difference between trajectory points, reflecting the change in curvature of the trajectory, and the curvature change of a smooth trajectory is smaller.

[0133] The path scoring module scores the candidate trajectories by comprehensively scoring mechanisms such as collision risk, traffic rule constraint and trajectory smoothness, and selects the optimal scheme. In a dynamic and complex environment, the safety and rationality of the path trajectory can be ensured, which is an important link in the entire path planning framework.

[0134] To sum up, the embodiment of the application obtains the multi-view camera image and the laser radar point cloud of the vehicle; determines the bird's eye view perception information of the vehicle according to the multi-view camera image and the laser radar point cloud; processes the bird's eye view perception information through the trajectory generation module to obtain multiple candidate path trajectories of the vehicle; processes the bird's eye view perception information and the multiple candidate path trajectories through the driving area prediction module to obtain a future driving area of the vehicle; processes the bird's eye view perception information and the multiple candidate path trajectories through the environment exploration module to obtain a future driving environment of the vehicle; and processes the future driving area and the future driving environment through the path scoring module to obtain a scoring result of the multiple candidate path trajectories. Through the embodiment of the application, path planning can be performed without relying on a high-precision map, the cost is low, and the embodiment has wider scene applicability.

[0135] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0136] Corresponding to the path planning method described in the above embodiment, Figure 5 An embodiment structure diagram of a path planning device provided by the embodiment of the application is shown.

[0137] In the embodiment, a path planning device can include:

[0138] The data acquisition module 501 is configured to acquire a multi-view camera image and a laser radar point cloud of a vehicle.

[0139] The bird's eye view perception module 502 is configured to determine bird's eye view perception information of the vehicle according to the multi-view camera image and the laser radar point cloud.

[0140] The trajectory generation module 503 is configured to process the bird's eye view perception information to obtain multiple candidate path trajectories of the vehicle.

[0141] The driving area prediction module 504 is configured to process the bird's eye view perception information and the multiple candidate path trajectories to obtain a future driving area of the vehicle.

[0142] The environment exploration module 505 is configured to process the bird's eye view perception information and the multiple candidate path trajectories to obtain a future driving environment of the vehicle.

[0143] The path scoring module 506 is configured to process the future driving area and the future driving environment by the path scoring module to obtain scoring results of the multiple candidate path trajectories.

[0144] In a specific implementation manner of the embodiment of the present application, the bird's eye view perception module can be specifically configured to: perform feature extraction on the multi-view camera image to obtain camera image features of the vehicle; perform feature extraction on the laser radar point cloud to obtain laser radar point cloud features of the vehicle; perform bird's eye view conversion on the camera image features to obtain camera image bird's eye view features of the vehicle; perform bird's eye view conversion on the laser radar point cloud features to obtain laser radar point cloud bird's eye view features of the vehicle; perform feature fusion on the camera image bird's eye view features and the laser radar point cloud bird's eye view features to obtain fusion features of the vehicle; decode the fusion features of the vehicle to obtain a bird's eye view map and target detection information of the vehicle; and determine bird's eye view perception information of the vehicle according to the bird's eye view map and the target detection information.

[0145] In a specific implementation manner of the embodiment of the present application, the trajectory generation module can be specifically configured to: generate multiple candidate path heat maps according to the bird's eye view perception information; wherein each candidate path heat map is used to represent a probability of each grid as a path point; and perform path point extraction in the multiple candidate path heat maps according to the probability of each grid as a path point to obtain multiple candidate path trajectories of the vehicle.

[0146] In a specific implementation manner of the embodiment of the present application, the driving area prediction module can include a spatial gating recurrent unit; and the driving area prediction module can be specifically configured to: perform time step-by-time step iterative processing on the bird's eye view perception information and the multiple candidate path trajectories by the spatial gating recurrent unit to obtain a future driving area of the vehicle; wherein the spatial gating recurrent unit is used to dynamically update a hidden state in a spatial dimension, and fuse input features and historical state information.

[0147] In a specific implementation manner of the embodiment of the present application, the environment exploration module can be specifically configured to: determine camera data and laser radar data corresponding to the bird's eye view perception information; perform multi-scale deformable attention mechanism processing on the camera data according to the multiple candidate path trajectories to obtain first extracted features; perform multi-layer perception processing on the laser radar data according to the multiple candidate path trajectories to obtain second extracted features; perform feature splicing processing on the first extracted features and the second extracted features to obtain spliced features; and perform multi-layer perception processing on the spliced features to obtain a future driving environment of the vehicle.

[0148] In a specific implementation process of the embodiment of the present application, the path scoring module can be specifically configured to: for the mth candidate path trajectory in the plurality of candidate path trajectories, respectively determine a sub-index score of the mth candidate path trajectory according to the future driving area and the future driving environment, wherein 1≤m≤M, M is the number of the plurality of candidate path trajectories; and determine a score result of the mth candidate path trajectory according to the sub-index scores of the mth candidate path trajectory.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0150] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0151] Figure 6 A schematic block diagram of an electronic device provided by an embodiment of the present application is shown, and only the parts related to the embodiments of the present application are shown for the convenience of description.

[0152] As Figure 6 shown, the electronic device 6 of this embodiment includes a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. The processor 60 implements the steps in each of the above path planning method embodiments when executing the computer program 62, such as Figure 2 steps S201 to S206 shown. Alternatively, the processor 60 implements the functions of each module / unit in the above device embodiments when executing the computer program 62, such as Figure 5 the functions of the modules 501 to 506 shown.

[0153] For example, the computer program 62 can be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 62 in the electronic device 6.

[0154] The electronic device 6 can include but is not limited to a vehicle-mounted computing device, a tablet computer, a desktop computer, a notebook computer, a palm computer, a server, etc. Those skilled in the art can understand, Figure 6The electronic device 6 is merely an example and does not limit the electronic device 6, which can include more or less components than shown, or combine some components, or different components, for example, the electronic device 6 can also include an input / output device, a network access device, a bus, etc.

[0155] The processor 60 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor.

[0156] The memory 61 can be an internal storage unit of the electronic device 6, such as a hard disk or a memory of the electronic device 6. The memory 61 can also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 61 can include both an internal storage unit and an external storage device of the electronic device 6. The memory 61 is used to store the computer program and other programs and data required by the electronic device 6. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit or module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit or module are only for easy distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0158] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0159] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0160] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / electronic device and method can be implemented in other ways. For example, the apparatus / electronic device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division, and there can be another division in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0161] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0162] In addition, each of the function units in each of the embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0163] The integrated module / unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be implemented by a computer program instructing related hardware to complete, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer readable storage medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable storage medium does not include electric carrier signals and telecommunication signals.

[0164] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A path planning method characterized by, The method comprises: acquiring a multi-view camera image and a laser radar point cloud of a vehicle; determining bird's eye view perception information of the vehicle according to the multi-view camera image and the laser radar point cloud; generating a plurality of candidate path heat maps according to the bird's eye view perception information; wherein each candidate path heat map is used to represent the probability of each grid as a path point; according to the probability of each grid as a path point, path point extraction is performed in the plurality of candidate path heat maps to obtain a plurality of candidate path trajectories of the vehicle; performing time step-by-time step iterative processing on the bird's eye view perception information and the plurality of candidate path trajectories by a spatial gate recurrent unit to obtain a future driving area of the vehicle; wherein the spatial gate recurrent unit is used to dynamically update a hidden state in a spatial dimension, fuse input features and historical state information; determining camera data and laser radar data corresponding to the bird's eye view perception information; performing multi-scale deformable attention mechanism processing on the camera data according to the plurality of candidate path trajectories to obtain first extracted features; performing multi-layer perception processing on the laser radar data according to the plurality of candidate path trajectories to obtain second extracted features; performing feature splicing processing on the first extracted features and the second extracted features to obtain spliced features; performing multi-layer perception processing on the spliced features to obtain a future driving environment of the vehicle; processing the future driving area and the future driving environment by a path scoring module to obtain a scoring result of the plurality of candidate path trajectories.

2. The path planning method of claim 1, wherein, The method comprises: extracting features from the multi-view camera image to obtain camera image features of the vehicle; extracting features from the laser radar point cloud to obtain laser radar point cloud features of the vehicle; performing bird's eye view conversion on the camera image features to obtain camera image bird's eye view features of the vehicle; performing bird's eye view conversion on the laser radar point cloud features to obtain laser radar point cloud bird's eye view features of the vehicle; fusing the camera image bird's eye view features and the laser radar point cloud bird's eye view features to obtain fusion features of the vehicle; decoding the fusion features of the vehicle to obtain a bird's eye map and target detection information of the vehicle; determining the bird's eye view perception information of the vehicle according to the bird's eye map and the target detection information.

3. The path planning method of any one of claims 1-2, wherein, The method comprises: for the mth candidate path trajectory in the plurality of candidate path trajectories, determining respective sub-index scores of the mth candidate path trajectory according to the future driving area and the future driving environment; wherein 1≤m≤M, and M is the number of the plurality of candidate path trajectories; determining a scoring result of the mth candidate path trajectory according to the respective sub-index scores of the mth candidate path trajectory.

4. The path planning method of claim 3, wherein, The method comprises: respectively determine a collision risk score, a traffic rule constraint score and a trajectory smoothness score of the mth candidate path trajectory; the score result of the mth candidate path trajectory is determined according to the respective sub-index scores of the mth candidate path trajectory, including: the collision risk score, the traffic rule constraint score and the trajectory smoothness score of the mth candidate path trajectory are summed to obtain the score result of the mth candidate path trajectory.

5. A route planning apparatus characterized by comprising: comprising: a data acquisition module configured to acquire multi-view camera images and laser radar point clouds of a vehicle; an aerial view perception module configured to determine aerial view perception information of the vehicle according to the multi-view camera images and the laser radar point clouds; a trajectory generation module configured to generate a plurality of candidate path heat maps according to the aerial view perception information; wherein each candidate path heat map is used to represent a probability of each grid as a path point; and path points are extracted in the plurality of candidate path heat maps according to the probability of each grid as a path point, to obtain a plurality of candidate path trajectories of the vehicle; a driving area prediction module configured to perform time step-by-time step iterative processing on the aerial view perception information and the plurality of candidate path trajectories by a spatial gating recurrent unit, to obtain a future driving area of the vehicle; wherein the spatial gating recurrent unit is configured to dynamically update a hidden state in a spatial dimension, to fuse input features and historical state information; an environment exploration module configured to determine camera data and laser radar data corresponding to the aerial view perception information; to perform multi-scale deformable attention mechanism processing on the camera data according to the plurality of candidate path trajectories, to obtain first extraction features; to perform multi-layer perception processing on the laser radar data according to the plurality of candidate path trajectories, to obtain second extraction features; to perform feature splicing processing on the first extraction features and the second extraction features, to obtain spliced features; and to perform multi-layer perception processing on the spliced features, to obtain a future driving environment of the vehicle; a path scoring module configured to process the future driving area and the future driving environment by a path scoring module, to obtain score results of the plurality of candidate path trajectories.

6. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. The computer program, when executed by a processor, implements the steps of the path planning method of any one of claims 1 to 4.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the path planning method of any one of claims 1 to 4.

Images