An end-to-end autonomous driving behavior planning method based on graph attention

Abstract

The application discloses an end-to-end automatic driving behavior planning method based on graph attention, and belongs to the technical field of automatic driving. The application mainly comprises the following steps: 1. preliminary extraction of multi-modal features; 2. implementation of multi-type perception tasks; 3. construction of features into a graph form according to perception results, and realization of interested feature screening; 4. feature processing by using a graph attention neural network; 5. prediction of a future driving path point of an automatic driving vehicle; and 6. behavior planning of the automatic driving vehicle. The end-to-end automatic driving behavior planning method based on the application introduces features in a graph form, conforms to a discrete information distribution mode in an outdoor automatic driving scene, explicitly uses perception results, and to some extent, makes up for the problem that a traditional end-to-end model lacks explainability.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, specifically a graph attention-based end-to-end autonomous driving behavior planning method. Background Technology

[0002] Autonomous driving technology encompasses a complete technology chain. While various perception algorithms have reached a high level of sophistication, the corresponding planning algorithms still primarily rely on traditional approaches based on finite state machines (FSMs) and decision trees. Currently, the rapid development of deep learning algorithms, the general improvement in computing power, and the easier acquisition of data have led to the exploration of end-to-end solutions for autonomous driving planning tasks. End-to-end solutions offer unique advantages over modular cascaded designs: 1) they avoid cascading errors and improve the representation of visual information; 2) as behavior planning is the ultimate problem in autonomous driving, end-to-end solutions can directly focus on the results, optimizing terminal performance and user experience. However, for autonomous driving tasks, vehicle safety is paramount. For planning models that directly control vehicle behavior, end-to-end algorithms, due to their lack of interpretability, still face many technical challenges before practical application.

[0003] For autonomous vehicles, perception models provide rich scene information, which planning algorithms rely on to control the vehicle. Therefore, the proper use of perception information is a prerequisite for safe driving. How to efficiently apply perception results to planning tasks is a pressing technical challenge. Most existing end-to-end autonomous driving behavior planning schemes use perception information implicitly, which is difficult to debug and iterate. In contrast, explicit application of perception results is not only more intuitive but also better conveys effective information to downstream tasks.

[0004] In real-world autonomous driving scenarios, foreground objects, crucial for vehicle path planning, are often distributed discretely within the scene. The semantic information contained within these foreground objects and their relative positions provide key references for behavior planning. Therefore, effectively utilizing this key foreground information to achieve path planning becomes the core of the problem. Given this discrete distribution of foreground objects, graphs, as a special data format, contain information within nodes and edges, perfectly meeting the needs of autonomous driving scenarios. Summary of the Invention

[0005] The purpose of this invention is to overcome the lack of interpretability in traditional end-to-end autonomous driving behavior planning algorithms. It proposes an end-to-end autonomous driving behavior planning method and system based on graph attention, which can explicitly utilize perception algorithms to capture scene information, enabling the model to focus on foreground objects that have key guidance for behavior planning, thereby achieving safe autonomous driving. It also provides a new paradigm for end-to-end autonomous driving behavior planning.

[0006] The technical solution adopted in this invention is as follows:

[0007] Let the variable subscripts "img" and "pc" represent camera image data and LiDAR point cloud data, respectively; let the variable subscripts "seg" and "det" represent semantic segmentation task and object detection task, respectively; let the variable subscript "node" represent graph form; and let the variable subscripts "wp" and "ctrl" represent predicted waypoints and behavior control, respectively.

[0008] An end-to-end autonomous driving behavior planning method based on graph attention includes the following steps:

[0009] Step (1). Acquire multimodal data collected by the camera and lidar, and extract image features and point cloud features respectively;

[0010] Step (2). Perform semantic segmentation and target detection on the image features and point cloud features respectively to obtain the semantic segmentation results for the image and the target detection results for the point cloud;

[0011] Step (3). Based on the semantic segmentation results and target detection results obtained in step (2), the image features and point cloud features extracted in step (1) are mapped into the graph form of the image modality and the graph form of the point cloud modality, respectively, to achieve the filtering of features of interest;

[0012] Step (4). Initialize the learnable query vector for path prediction, and use a graph attention neural network to perform feature aggregation on the graph form of the image modality and the graph form of the point cloud modality obtained in step (3), and update the learnable query vector for path prediction.

[0013] Step (5). Encode the relative position between the vehicle's current position and the target position into a high-dimensional feature. Use the updated learnable query vector for path prediction and the encoded high-dimensional feature as input to the recurrent neural network, and output a series of predicted path points from the vehicle's current position to the target position.

[0014] Step (6). Transform a series of predicted path points from the current position to the target position into the vehicle's underlying control signals to complete the autonomous vehicle behavior planning task.

[0015] Further, step (1) includes:

[0016] Step (1.1). Acquire camera image data and generate image features through an image feature extraction network;

[0017] Step (1.2). Obtain lidar point cloud data. First, normalize the irregular point cloud to voxel space, and then generate point cloud features through a point cloud feature extraction network.

[0018] Furthermore, step (2) includes:

[0019] Step (2.1). Input the image features into the semantic segmentation output head to obtain the semantic segmentation result for the image; the semantic segmentation result includes the category information of each pixel in the image;

[0020] Step (2.2). Input the point cloud features into the target detection output head to obtain the target detection result for the point cloud; the target detection result includes the target category and location.

[0021] Furthermore, the semantic segmentation output head and the object detection output head are implemented using two independent multilayer perceptrons.

[0022] Furthermore, step (3) includes:

[0023] Step (3.1). Divide the image into non-overlapping image blocks. Each pixel corresponds to a semantic segmentation result. Select image blocks in which more than 50% of the semantic segmentation results belong to the foreground as effective nodes. Based on the position of the image block corresponding to the effective node in the whole image, obtain the local image features corresponding to the effective node. Perform average pooling on the pixel dimension of the local image features to generate effective node features. Record the positional relationship between effective nodes using an adjacency matrix to obtain the graph form of the initial image modality.

[0024] Step (3.2). Perform global average pooling on the image features to obtain the global features of the image modality, and connect them as an effective node to each effective node generated in step (3.1). Update the adjacency matrix in step (3.1) to obtain the updated graph form of the image modality.

[0025] Step (3.3). Determine the position of the center of the foreground object obtained by target detection in three-dimensional space, filter the voxels corresponding to the foreground object according to the position, use them as effective nodes, and use the point cloud features corresponding to the foreground as effective node features. The positional relationship between effective nodes is recorded using an adjacency matrix to obtain the graph form of the initial point cloud mode.

[0026] Step (3.4). Perform global average pooling on the point cloud features to obtain the global features of the point cloud modality, and connect it as an effective node to each effective node generated in step (3.3). Update the adjacency matrix in step (3.3) to obtain the updated graph form of the point cloud modality.

[0027] Furthermore, the positional relationships between the effective nodes are recorded using an adjacency matrix, including:

[0028] For an image, edges are determined based on the positional relationship of valid nodes in the image. If the image patches corresponding to two valid nodes are adjacent in the whole image, then the two valid nodes are connected.

[0029] For point clouds, a distance threshold is set, and each valid node will be connected to other valid nodes within that threshold distance.

[0030] The connected edges are recorded using two adjacency matrices.

[0031] Furthermore, the graph attention neural network includes a first graph convolutional network encoder, a second graph convolutional network encoder, and a Transformer decoder.

[0032] Furthermore, step (4) includes:

[0033] Step (4.1). Input the image modality in graph form into the first graph convolutional network encoder, and use the encoding result as the first key vector and the first value vector related to the environment information;

[0034] Step (4.2). Input the graph form of the point cloud modality into the second graph convolutional network encoder, and use the encoding result as the second key vector and second value vector related to the environment information;

[0035] Step (4.3). Initialize a learnable query vector for path prediction, concatenate the first key vector and the second key vector, and concatenate the first value vector and the second value vector. Use the learnable query vector, the concatenated key vector, and the concatenated value vector as input to the Transformer decoder, and output the updated learnable query vector for path prediction.

[0036] Furthermore, in step (5), a multilayer perceptron is used to encode the relative position between the vehicle's current position and the target position as a high-dimensional feature.

[0037] Furthermore, the underlying control signals of the vehicle include accelerator, brake, and steering commands.

[0038] The beneficial effects of this invention are:

[0039] (1) This invention proposes a complete end-to-end autonomous driving behavior planning method based on graph attention. It introduces graph data form and uses perception results to construct the features corresponding to the foreground objects that play a key role in the final behavior planning into the form of graph, so that the model can focus on effective information and obtain high-quality planning results.

[0040] (2) This invention utilizes prior knowledge to effectively extract relatively sparse foreground features from dense scene information while preserving their spatial positional relationships. It relies on the powerful performance of graph attention neural networks to achieve global information extraction for path planning.

[0041] (3) By explicitly utilizing perception information, this invention makes up for the lack of interpretability of end-to-end models to a certain extent, and provides a new paradigm for end-to-end autonomous driving behavior planning tasks, creating new possibilities for subsequent related research and industrial implementation in this field. Attached Figure Description

[0042] Figure 1 This is a framework diagram of the end-to-end autonomous driving behavior planning method based on graph attention proposed in this invention. Detailed Implementation

[0043] The present invention will be further described below with reference to the accompanying drawings. The drawings are merely illustrative diagrams of the present invention. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0044] like Figure 1 As shown, the specific implementation steps of the end-to-end autonomous driving behavior planning method based on graph attention of the present invention include:

[0045] Step (1). Input the multimodal data collected by the camera and LiDAR, and use independent deep learning networks to perform preliminary feature extraction, specifically:

[0046] Step (1.1). Input the image data I captured by the camera. img Through a convolution-based image feature extraction network Φ img Generate image features F img :

[0047] F img =Φ img (I img )

[0048] Step (1.2). Input the point cloud data I collected by the lidar. pc For point cloud data, its disorder and irregularity are not conducive to neural network processing, so it is necessary to first normalize the irregular point cloud to voxel space. Then, a point cloud feature extraction network Φ based on sparse convolution is used. pc Generate point cloud features:

[0049]

[0050] Among them, the image features generated in step (1.1) H×W corresponds to the original image resolution after cropping; the point cloud features generated in step (1.2) X×Y×Z correspond to the horizontal and vertical directions in three-dimensional space, which can be mapped to a real space range of 102.4m×102.4m×8m. The feature dimension C of both is set to 256. This indicates that the point cloud is normalized to the voxel space.

[0051] The above is from image data I img Image features F converted to pixel level img Point cloud data I pc Point cloud features F converted to voxel level pc These describe the two-dimensional and three-dimensional spatial environment information of the current scene, respectively. These features will be used in the next step for the corresponding perception task. Step (2). Based on the image features F obtained in step (1), img and point cloud features F pc The multimodal features are used to infer semantic segmentation and object detection results for the image and point cloud, respectively, through semantic segmentation and object detection output heads.

[0052] Step (2.1). Extract image features F img Input the semantic segmentation output header to obtain the image semantic segmentation result O. seg :

[0053]

[0054] Step (2.2). Transfer the point cloud features F pc Input the target detection output header to obtain the point cloud target detection result O det :

[0055]

[0056] Among them, O seg and O det These represent pixel-level semantic segmentation results and 3D object detection results, respectively. seg Includes category information for each pixel, O det It includes information such as the detected target's three-dimensional spatial location, shape, angle, and category.

[0057] In this embodiment, the semantic segmentation output head and the object detection output head are implemented using MLP.

[0058] Step (3). Based on the perception results obtained in step (2), and combined with prior knowledge, select the foreground features of interest from the environmental information features generated in step (1), and use them as the node features F of the graph. node Simultaneously, its spatial location information is preserved and recorded in the adjacency matrix A. To retain global feature information, global average pooling is performed on the original image features and the original point cloud features respectively, resulting in global feature F. g It is added to the corresponding graph.

[0059] Specifically, based on the semantic segmentation results and object detection results obtained in step (2), the features extracted in step (1) are remapped into a graph form to achieve feature selection of interest, including:

[0060] Step (3.1). Transfer image I img Divided into Each image patch consists of non-overlapping image blocks, where p controls the number of pixels in each image patch. p can be set to 8 or 16, depending on the resolution of the original image. After segmentation, each image patch contains p × p pixels, and each pixel corresponds to a semantic segmentation result. In this embodiment, objects such as vehicles and pedestrians are identified as foreground, and buildings and the sky are identified as background. Image patches with more than 50% foreground pixels are selected as effective nodes. Based on the position of the image patch corresponding to the effective node in the entire image, the local image features corresponding to the effective node are obtained. The local image features are then averaged and pooled along the pixel dimension to generate the effective node features F. inode After filtering out all valid nodes, edges are determined based on the positional relationships of the valid nodes in the image. If the image patches corresponding to two valid nodes are adjacent in the entire image, then these two valid nodes will be connected. The positional relationships between valid nodes are determined using an adjacency matrix. Records, including This represents the total number of image patches. Specifically, when the i-th node and the j-th node are adjacent, A img [i][j]=A img [j][i] = 1.

[0061] Step (3.2). Perform global average pooling on the image features to obtain the global features of the image modalities. And connect it as a valid node to each valid node generated in step (3.1), forming an adjacency matrix. Expanded to

[0062] Step (3.3). Determine the position of the center of the foreground object obtained from target detection in 3D space, filter out the voxels corresponding to the foreground object based on the position, use them as effective nodes, and use the point cloud features corresponding to the foreground as effective node features F. pnode The positional relationships between valid nodes are determined using an adjacency matrix. The record is set, where m = X × Y × Z represents the total prime number after the irregular point cloud is normalized to the voxel space. Given the sparsity of foreground targets in the point cloud scene, a distance threshold θ = 5m is set, and each valid node will be connected to other valid nodes within this threshold distance.

[0063] Step (3.4). Perform global average pooling on the point cloud features to obtain the global modal features of the point cloud. And connect it as a valid node to each valid node generated in step (3.3), forming an adjacency matrix. It is expanded to m′=X×Y×Z+1.

[0064] The feature {F} generated in step (3) above node A} represents the graph-like features of interest selected by combining the perception results in this invention, which are used for subsequent path prediction.

[0065] Step (4). The graph attention neural network contains an encoder consisting of multiple layers of graph convolutional networks. and decoders based on the Transformer architecture The encoder performs feature aggregation and processing on the graph-like features obtained in step (3), using them as keys and values ​​related to environmental information; it initializes a learnable query for path prediction, and the decoder uses the keys and values ​​related to environmental information to learn and update the path prediction query, specifically:

[0066] Step (4.1). Input the image features in graph form generated in step (3) into the graph convolutional network encoder. In this context, the key and value related to environmental information are encoded and denoted as K. img With V img :

[0067]

[0068] Step (4.2). Input the point cloud features in graph form generated in step (3) into the graph convolutional network encoder. In this context, the key and value related to environmental information are encoded and denoted as K. pc With V pc :

[0069]

[0070] Step (4.3). Initialize a learnable feature vector. As a path prediction query, the key and value related to the environmental information obtained in steps (4.1) and (4.2) are concatenated and fed into the decoder based on the Transformer structure. The system learns and updates path prediction queries:

[0071]

[0072] Among them, Q pred This query represents the updated future path prediction. This indicates an initial query used for future path prediction.

[0073] Step (5). The path prediction recurrent neural network is used as the output head of the path prediction. The offset of the target position relative to the current position of the vehicle is used to encode the target relative position as a high-dimensional feature, which is used as the initial hidden state of the recurrent neural network. The query Q updated in step (4) is used for path prediction. pred As input, complete the path planning from the current location to the target location, as follows:

[0074] Step (5.1). Encode the target's relative position (x, y) into a high-dimensional feature H using a multilayer perceptron:

[0075] H = MLP(x,y)

[0076] Step (5.2). Update the path prediction Q from step (4). pred As input, the path prediction recurrent neural network outputs a series of predicted path points from the vehicle's current position to the target position:

[0077]

[0078] Where H is the initial hidden state of the path prediction recurrent neural network, and O wp={(x1,y1),(x2,y2),(x3,y3),…} represents the position offset relative to the previous time step, which is considered as a series of predicted path points for the vehicle to travel from its current position to the target position. In this embodiment, the path prediction recurrent neural network can be an RNN, LSTM, or GRU.

[0079] During the training phase, camera images, LiDAR point clouds, and real path points are acquired as samples. The camera images and LiDAR point clouds are used as inputs, and the above model is trained based on the error between the predicted path points and the real path points.

[0080] Step (6). The behavior conversion controller γ converts the series of predicted path points obtained in step (5) into the underlying control signals of the vehicle, thus completing the autonomous vehicle behavior planning task:

[0081] O ctrl =γ(O wp )

[0082] Among them, the behavior conversion controller is a proportional-derivative-integral controller, which records the current state in real time and converts the future path points generated in step (5) into the vehicle's underlying control signals. ctrl It includes accelerator, brake, and steering commands, and is a control signal used to implement behavior planning for autonomous vehicles.

[0083] The above description is merely a specific embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-described technical features, but should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the concept of this application. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. An end-to-end autonomous driving behavior planning method based on graph attention, characterized by the following steps: Step (1). Acquire multimodal data collected by the camera and lidar, and extract image features and point cloud features respectively; Step (2). Perform semantic segmentation and target detection on the image features and point cloud features respectively to obtain the semantic segmentation results for the image and the target detection results for the point cloud; Step (3). Based on the semantic segmentation results and target detection results obtained in step (2), the image features and point cloud features extracted in step (1) are mapped into the graph form of the image modality and the graph form of the point cloud modality, respectively, to achieve the filtering of features of interest; Step (4). Initialize the learnable query vector for path prediction, and use a graph attention neural network to perform feature aggregation on the graph form of the image modality and the graph form of the point cloud modality obtained in step (3), and update the learnable query vector for path prediction. Step (5). Encode the relative position between the vehicle's current position and the target position into a high-dimensional feature. Use the updated learnable query vector for path prediction and the encoded high-dimensional feature as input to the recurrent neural network, and output a series of predicted path points from the vehicle's current position to the target position. Step (6). Transform a series of predicted path points from the current location to the target location into the vehicle's underlying control signals to complete the autonomous vehicle behavior planning task.

2. The end-to-end autonomous driving behavior planning method based on graph attention according to claim 1, characterized in that, Step (1) includes: Step (1.1). Acquire camera image data and generate image features through an image feature extraction network; Step (1.2). Obtain lidar point cloud data. First, normalize the irregular point cloud to voxel space, and then generate point cloud features through a point cloud feature extraction network.

3. The end-to-end autonomous driving behavior planning method based on graph attention according to claim 1, characterized in that, Step (2) includes: Step (2.1). Input the image features into the semantic segmentation output head to obtain the semantic segmentation result for the image; the semantic segmentation result includes the category information of each pixel in the image; Step (2.2). Input the point cloud features into the target detection output head to obtain the target detection result for the point cloud; the target detection result includes the target category and location.

4. The end-to-end autonomous driving behavior planning method based on graph attention according to claim 3, characterized in that, The semantic segmentation output head and the object detection output head are implemented using two independent multilayer perceptrons.

5. The end-to-end autonomous driving behavior planning method based on graph attention according to claim 1, characterized in that, Step (3) includes: Step (3.1). Divide the image into non-overlapping image blocks. Each pixel corresponds to a semantic segmentation result. Select image blocks in which more than 50% of the semantic segmentation results belong to the foreground as effective nodes. Based on the position of the image block corresponding to the effective node in the whole image, obtain the local image features corresponding to the effective node. Perform average pooling on the pixel dimension of the local image features to generate effective node features. Record the positional relationship between effective nodes using an adjacency matrix to obtain the graph form of the initial image modality. Step (3.2). Perform global average pooling on the image features to obtain the global features of the image modality, and connect them as an effective node to each effective node generated in step (3.1). Update the adjacency matrix in step (3.1) to obtain the updated graph form of the image modality. Step (3.3). Determine the position of the center of the foreground object obtained by target detection in three-dimensional space, filter the voxels corresponding to the foreground object according to the position, use them as effective nodes, and use the point cloud features corresponding to the foreground as effective node features. The positional relationship between effective nodes is recorded using an adjacency matrix to obtain the graph form of the initial point cloud mode. Step (3.4). Perform global average pooling on the point cloud features to obtain the global features of the point cloud modality, and connect it as an effective node to each effective node generated in step (3.3). Update the adjacency matrix in step (3.3) to obtain the updated graph form of the point cloud modality.

6. The end-to-end autonomous driving behavior planning method based on graph attention according to claim 5, characterized in that, The positional relationships between the effective nodes are recorded using an adjacency matrix, including: For an image, edges are determined based on the positional relationship of valid nodes in the image. If the image patches corresponding to two valid nodes are adjacent in the whole image, then the two valid nodes are connected. For point clouds, a distance threshold is set, and each valid node will be connected to other valid nodes within that distance threshold. The connected edges are recorded using two adjacency matrices.

7. The end-to-end autonomous driving behavior planning method based on graph attention according to claim 1, characterized in that, The graph attention neural network includes a first graph convolutional network encoder, a second graph convolutional network encoder, and a Transformer decoder.

8. The end-to-end autonomous driving behavior planning method based on graph attention according to claim 7, characterized in that, Step (4) includes: Step (4.1). Input the image modality in graph form into the first graph convolutional network encoder, and use the encoding result as the first key vector and the first value vector related to the environment information; Step (4.2). Input the graph form of the point cloud modality into the second graph convolutional network encoder, and use the encoding result as the second key vector and second value vector related to the environment information; Step (4.3). Initialize a learnable query vector for path prediction, concatenate the first key vector and the second key vector, and concatenate the first value vector and the second value vector. Use the learnable query vector, the concatenated key vector, and the concatenated value vector as input to the Transformer decoder, and output the updated learnable query vector for path prediction.

9. The end-to-end autonomous driving behavior planning method based on graph attention according to claim 1, characterized in that, In step (5), a multilayer perceptron is used to encode the relative position between the vehicle's current position and the target position as a high-dimensional feature.

10. The end-to-end autonomous driving behavior planning method based on graph attention according to claim 1, characterized in that, The underlying control signals of the vehicle include accelerator, brake, and steering commands.

Images