End-to-end motion planning method fused with dynamic space attention mechanism

Abstract

The invention discloses an end-to-end motion planning method fused with a dynamic space attention mechanism, and aims to solve the problems that a traditional end-to-end motion planning method is single in sensing means, insufficient in interpretability and the like. The invention provides an end-to-end motion planning method of a double-branch design fusing a dynamic space attention mechanism, time sequence state information and track and control prediction. According to the method, a semantic aerial view is generated by using a surrounding visual image, the attention weight of a key object is dynamically adjusted according to a navigation instruction and a vehicle speed through a dynamic space attention mechanism, a time sequence state feature is constructed by combining historical multi-frame state information, and the semantic aerial view and the time sequence state feature are jointly input into a multi-branch network for track prediction and control prediction. Therefore, a safe and smooth planning track and a control signal are generated. Experimental results show that the method significantly improves the safety, generalization ability and interpretability of end-to-end motion planning, and provides an efficient solution for automatic driving in a complex scene.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and is an end-to-end motion planning method based on deep learning, specifically an end-to-end motion planning method for unmanned vehicles that integrates a dynamic spatial attention mechanism. Background Technology

[0002] Motion planning is a core indicator of the intelligence level of autonomous driving in unmanned vehicles. End-to-end autonomous driving methods directly map raw sensor data into trajectories or control signals, avoiding cascading errors caused by complex modular designs and manual rules, and have the advantages of high efficiency and simplicity. However, traditional end-to-end methods have safety risks due to their limited perception means (such as reliance on monocular images) and insufficient interpretability, especially in complex dynamic environments where they struggle to meet real-time and robustness requirements.

[0003] End-to-end planning based on Conditional Imitation Learning (CIL) offers potential solutions for known, partially known, and unknown environments. Early methods such as CILRS (an extension of CIL based on residual networks and velocity prediction) utilize monocular vision to predict control variables, while Trajectory-guided Control Prediction (TCP) achieves end-to-end planning through monocular images, showing some success. Surround vision Conditional Imitation Learning (SCIL) introduces multi-view cameras, and Dynamic Surround Vision Conditional Imitation Learning (DSCIL) further integrates multi-frame states, but it does not fully consider dynamic perception driven by navigation commands. Learning by Cheating (LBC) methods utilize privileged agents to train visual models, but rely on omniscient environmental information, resulting in limited generalization ability. Some studies generate interpretable representations (such as semantic bird's-eye views) through multi-task learning, but these often employ single-perception methods, limiting the model's adaptability in unknown scenarios. Summary of the Invention

[0004] The purpose of this invention is to propose an end-to-end motion planning method that integrates a dynamic spatial attention mechanism to improve the planning accuracy, real-time performance, and interpretability of autonomous vehicles in complex dynamic environments.

[0005] The technical solution adopted in this invention is as follows:

[0006] 1. Hierarchical Feature Acquisition and Combination: A semantic bird's-eye view (BEV) is generated using surround vision images. A dynamic spatial attention mechanism dynamically adjusts the attention weights for different areas of the BEV based on high-level navigation commands (straight, left turn, right turn, etc.), generating bird's-eye view features with additional spatial attention. Temporal state features are generated by combining multi-frame historical vehicle state information (speed, steering wheel angle). The bird's-eye view features and temporal state features are combined and used as input for subsequent networks.

[0007] 2. Trajectory and Control Dual-Branch Planning: Joint features are input into the trajectory prediction branch, which uses a GRU network to predict trajectory points for the next K timestamps and combines this with a PID control algorithm to generate trajectory control signals. Simultaneously, trajectory features guide the control prediction branch, fusing visual and bird's-eye view features to predict multi-step control signals (accelerator, brake, steering wheel angle). Finally, a scene-adaptive fusion strategy is used to fuse the trajectory control signals and control prediction signals to generate the final control information.

[0008] Step 1 uses six surround visual images from the vehicle's surroundings (front, left front, right front, rear, left rear, and right rear, with a resolution of 480×320 and a field of view of 90 degrees) as input, and extracts multi-view features Ft (i=1,...,6) through a pre-trained ResNet-101 backbone network. Subsequently, based on the BEV query Q∈

[0009] R^(H×W×C) (where H and W represent the spatial dimensions of the BEV plane, and C is the number of channels) generates a semantic bird's-eye view (BEV). The BEV feature Bt is fused with historical information from the previous timestamp Bt-1 through a temporal self-attention mechanism, and spatial information is extracted from Ft through a spatial cross-attention mechanism. After processing by a 6-layer encoder, a unified BEV representation is generated. A dynamic spatial attention mechanism is introduced to dynamically adjust the attention weight of the BEV region based on high-level navigation commands C (including straight, left turn, right turn, left lane change, right lane change, etc.) and the vehicle's current speed v. In the specific implementation, a multi-head attention mechanism is used to calculate the weights, where the attention score is driven by both navigation commands and speed. Priority is given to key objects (such as vehicles, pedestrians, traffic lights, and lane lines), while reducing the weight of background noise (such as distant sky or buildings), thereby generating a bird's-eye view feature Fb with additional spatial attention. Simultaneously, the vehicle's historical three-frame state information (including speed v, steering wheel angle α, sampling frequency of 2Hz, covering a 1.5-second time window) is input into a multilayer perceptron (MLP) encoding network to generate temporal state features Js. The MLP consists of two fully connected layers, each with 512 neurons, using the ReLU activation function. The temporal feature Js is further modeled using a GRU network (with 256 hidden layers) to capture the dynamic trend and the continuity of vehicle motion. Finally, the bird's-eye view feature Fb and the temporal feature Js are concatenated to generate a joint feature vector j.trj =MLP(Concat(F b J s This serves as the input for the subsequent trajectory prediction branch.

[0010] In step 2, the joint feature F is input to the trajectory prediction branch. A GRU network (with a hidden layer dimension of 256, a prediction step size K = 10, and a time window of 5 seconds) is used to predict trajectory points w = {wp1, ..., wpk} for the next K timestamps, where each trajectory point is represented by two-dimensional coordinates (x, y). During trajectory prediction, the GRU network takes the joint feature F as input, combines it with the hidden state of the previous timestamp, updates the state layer by layer, and outputs the trajectory points. The trajectory points w are used to calculate the expected velocity v. * Based on v * The difference between the current speed *v* and the initial values ​​of throttle *t* and brake *b* are calculated. The steering wheel angle *s* is determined by the angle difference between the trajectory point and the vehicle's current orientation. The trajectory point *w* is further input into the PID control algorithm to generate the trajectory control signal *a*. trj The joint feature F is simultaneously input into the control prediction branch, combined with the trajectory point w to transform it into trajectory feature Ft. This Ft is then fused with the visual feature Fc and the bird's-eye view feature Fb through a fully connected layer (512 neurons, ReLU activation) to generate the control prediction feature Jctl. Jctl is input into another GRU network (hidden layer dimension 256, prediction step size 4 steps) to predict the multi-step control signal a. ctl Finally, the trajectory control signal and control prediction signal are fused to generate the final control signal through a scene-adaptive fusion strategy. The fusion strategy is based on a scene weight β (range 0-0.5). In turning scenarios (e.g., left turn, right turn), β is set to 0.2 to prioritize the use of the control prediction signal and avoid PID tracking failure; in straight-ahead or start-stop scenarios, β is set to 0.5 to balance the contributions of the two branches. The final control signal is output to the vehicle controller to drive the vehicle's movement.

[0011] Compared with the prior art, the significant advantages of this invention are:

[0012] (1) A semantic bird's-eye view generation method integrating dynamic spatial attention mechanism is proposed. Combined with navigation instructions, the perception focus is optimized, which enhances the interpretability and security of end-to-end planning.

[0013] (2) By integrating temporal state information and dual-branch design (trajectory prediction and control prediction), the limitations of a single trajectory or control method are overcome, and the planning accuracy and real-time performance in complex scenarios are significantly improved.

[0014] (3) By optimizing the trajectory and control signal through the scene adaptive fusion strategy, the problems of PID tracking failure and dependence on the independent and identically distributed assumption are solved. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of a perception network architecture;

[0016] Figure 2 A schematic diagram of the network architecture for the trajectory prediction branch and the control prediction branch;

[0017] Figure 3 This is a schematic diagram of the trajectory guidance control network structure;

[0018] Figure 4 Visualization diagram of BEV weights under different navigation commands and speeds

[0019] Figure 5 To verify the experimental results Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings.

[0021] This invention proposes an end-to-end motion planning method that integrates a dynamic spatial attention mechanism. The system architecture is as follows: Figure 1 As shown, this method generates a semantic bird's-eye view by surrounding a visual image, and combines temporal state information and a dual-branch network to generate the vehicle's planned trajectory and control signals.

[0022] (1) Details of Perceptual Network Technology

[0023] like Figure 1 As shown, the perception network takes six surround visual images from the vehicle's surroundings as input: front, left front, right front, rear, left rear, and right rear. The resolution is 480×320, and the field of view is 90 degrees. Camera intrinsic and extrinsic parameters (including focal length, distortion coefficients, and extrinsic matrix) are obtained through pre-calibration. The ResNet-101 backbone network extracts multi-view features Ft (i = 1,...,6), with a feature dimension of 2048 for each view. The BEV query Q∈R^(200×200×64) is defined as 200×200 grid cells, each corresponding to a 0.1-meter range in the real world, centered and aligned with the vehicle's position. The initial BEV feature Bt-1 is saved from the previous timestamp, and the current timestamp's BEV feature Bt is generated through a 6-layer encoder.

[0024] Each encoder layer includes a temporal self-attention module and a spatial cross-attention module: temporal self-attention extracts historical information from Bt-1, and spatial cross-attention extracts multi-view spatial information from Ft. The attention mechanism adopts a multi-head structure (8 heads, 64 dimensions), and a feedforward network (1024 neurons in a fully connected layer, ReLU activation) further refines the features. The semantic segmentation module is based on an Encoder-Decoder structure, using ResNet-18 to extract feature layers, upsampling by 4 times and concatenating with low-level features to output a 200×200 semantic map. Categories include vehicles, roads, lane lines, pedestrians, and traffic lights. The loss function uses weighted cross-entropy loss, with weights calculated based on the pixel ratio of each category. The dynamic spatial attention module takes BEV features Bt, navigation instructions C, and normalized velocity v∈[0,1] as input, then increases the weight of specific appealing key objects and reduces the weight of background noise to generate enhanced bird's-eye view features Fb. In the semantic segmentation supervision results, the average pixel accuracy is 82.2%, and the average IoU is 55.3%, verifying the effectiveness of the perception module.

[0025] (2) Trajectory Prediction Branch Technique Details

[0026] As per the appendix of this invention Figure 2 As shown, the trajectory prediction branch takes the bird's-eye view feature Fb and the temporal state feature Js as input. The temporal state information is obtained from three historical frames of data (velocity v, steering wheel angle α, sampling frequency 2Hz, time window 1.5 seconds). The temporal state information is mapped to the initial feature S through a fully connected layer (256 neurons, ReLU activation), input to the GRU network (hidden layer dimension 256), and outputs the temporal feature Js. Joint feature j trj =MLP(Concat(F b J s )), j trj Inputting the data into a GRU network, predicting trajectory point w, with the trajectory loss using the L1 norm. Subsequently, trajectory control signal a is generated based on trajectory point w. trj .

[0027] (3) Details of Control Prediction Branch Techniques

[0028] like Figure 3 As shown, the control prediction branch is guided by trajectory point w, and trajectory features are extracted from w through a fully connected layer (128 neurons, ReLU activation) to generate trajectory features Fw and control prediction features j. trj The input is fed into a GRU network (hidden layer dimension 128, prediction step size 4, time interval 0.5 seconds) to predict a multi-step control signal a. ctl Loss is controlled using the L1 norm: in Let represent the actual control value and the predicted control value at time i, respectively. The control signal is guided by the trajectory point w, which enhances the responsiveness in dynamic scenarios.

[0029] (4) Output fusion module details

[0030] The trajectory control signal and the control prediction signal are fused according to weights β and (1-β) to generate the final control signal a = β·a trj +(1-β)a ctl The weight β is dynamically adjusted according to the scenario: β = 0.2 for turning scenarios (e.g., left turn, right turn, U-turn), and β = 0.5 for straight-ahead or stop-and-go scenarios (e.g., road tracking, parking), balancing the two branches; β = 0.3 for lane-changing scenarios (e.g., left lane change, right lane change), with a slight bias. Scene recognition is based on navigation command C and speed v, used to distinguish between straight-ahead and turning.

[0031] (5) Experimental verification of motion planning

[0032] Data collection and experimental verification were conducted using the CARLA 0.9.10 autonomous driving simulation platform. The collected data included surround-view images from the expert system during driving, covering six views (front, left front, right front, rear, left rear, and right rear), all with a resolution of 480×320 and a field of view of 90 degrees. The semantic bird's-eye view ground truth size was 200×200, representing environmental information within 20 meters of the vehicle. Simultaneously, vehicle speed, expert commands (accelerator, brake, steering angle), target points, advanced navigation commands (straight ahead, left turn, right turn, lane change, etc.), and camera intrinsic and extrinsic parameters were collected. Driving data was collected using the expert system on Town01, Town03, Town04, Town05, Town07, and Town10 of the CARLA simulation platform as training datasets. The data acquisition frequency was 2fps, covering various typical autonomous driving test scenarios, including basic urban areas, highways, complex city centers, rural roads, and high-precision map cities. The training dataset contains 180k frames. Town05 was selected as the test scenario for evaluating the model and was not used for training.

[0033] The Town05 benchmark includes various navigation tasks: straight driving, turning, lane changing, and intersection navigation, with scenarios containing dynamic obstacles (such as pedestrians and vehicles) and static obstacles (such as traffic lights and roadblocks). Test conditions cover six weather conditions: four known weather conditions (ClearNoon, WetNoon, HardRainNoon, ClearSunset) and two unknown weather conditions (WetSunset, SoftRainSunset). Task success is defined as the vehicle reaching the destination within a limited time; exceeding a collision threshold (e.g., a severe collision with a pedestrian or vehicle) is considered a failure. Metrics include Driving Score (DS), Route Completion Ratio (RC), and Violation Score (IS, ranging from 0 to 1, higher is better).

[0034] Experimental results are as follows Figure 5 As shown, CSATS performs excellently in complex scenarios in Town05 (such as intersection navigation and dense dynamic obstacles). Comparative experiments are shown below. Figure 5 As shown, CSATS achieved a DS of 74.09, RC of 83.25%, and IS of 0.89, outperforming TCP (DS 64.88, RC 82.13%, IS 0.79), TLSSCIL (DS 68.56, RC 80.66%, IS 0.85), and Transfuser (DS 65.00, RC 84.42%, IS 0.77). Under unknown weather conditions (such as a wet sunset), CSATS improved DS by 12% (compared to TLSSCIL) and IS by 0.04 (compared to Transfuser), indicating that the dynamic spatial attention mechanism and temporal state information effectively improved the model's generalization ability.

Claims

1. An end-to-end motion planning method incorporating a dynamic spatial attention mechanism, characterized in that, Includes the following steps: 1) Generate a semantic bird's-eye view using surround vision images. Adjust the regional weights of the semantic bird's-eye view based on advanced navigation instructions and vehicle speed using a dynamic spatial attention mechanism to generate a bird's-eye view feature vector. Combine this with multi-frame historical vehicle state information to generate a temporal state feature vector. Combine the bird's-eye view feature vector and the temporal state feature vector to generate a joint feature vector. 2) Input the joint feature vector into the trajectory prediction branch and the control prediction branch. Use the trajectory prediction branch to generate future trajectory points and the control prediction branch to generate multi-step control signals. Combine the outputs of the trajectory prediction branch and the control prediction branch using a scene adaptive fusion strategy to generate the vehicle's final control signal.

2. According to step 1) of claim 1, the implementation details are as follows: using six surround visual images as input, multi-view features are extracted through a ResNet-101 backbone network, an M×M pixel semantic bird's-eye view is generated using BEV query, covering a range of M / 10 meters around the vehicle, a dynamic spatial attention mechanism calculates attention based on navigation instructions and vehicle speed, and generates a bird's-eye view feature vector Fb, and historical multi-frame state information is encoded through a GRU network to generate a temporal state feature vector j. s Trajectory features are obtained by splicing operations.

3. According to step 2) of claim 1, the implementation details are as follows: the trajectory prediction branch uses the GRU network to predict trajectory points for the next four timestamps, and the control prediction branch uses the trajectory points as a guide to predict multi-step control signals using the GRU network, including accelerator, brake and steering wheel angle. The weights are dynamically adjusted according to the scene type through the scene adaptive fusion strategy, and the outputs of the trajectory prediction branch and the control prediction branch are fused.

4. The dynamic spatial attention mechanism according to claim 2 is implemented as follows: navigation instructions are mapped to a 9-dimensional one-hot encoded vector, vehicle speed is normalized to 0 to 1, task-driven features are generated through a fully connected layer, the number of heads in the multi-head attention mechanism is 8, the dimension is 64, attention scores are calculated, and the weights of vehicles, pedestrians, traffic lights and lane lines are prioritized, while the weights of non-critical areas, such as background areas, are reduced.

5. The timing state information processing according to claim 2, in detail as follows: the implementation of the timing state feature vector includes: Three frames of vehicle history state information, including speed and steering wheel angle, are collected at a sampling frequency of 2Hz and a time window of 1.5 seconds. The data are mapped to a 256-dimensional initial feature vector through a fully connected layer, activated by the ReLU function, and then input into a GRU network. The GRU network has a hidden layer dimension of 256 and two layers.

6. According to claim 3, the dual-branch prediction method, for the trajectory prediction branch, the implementation details are as follows: The bird's-eye view feature vector Fb and the temporal state feature vector j... s Trajectory features j are obtained by concatenation operation. trj As shown in formula (1), j trj The data is then input into the trajectory GRU unit. The trajectory GRU network prediction step size is 4, the time window is 2 seconds, the time interval is 0.5 seconds, and the trajectory points are represented as two-dimensional coordinates. j trj =MLP(Concat(F b ,J s )) (1) The trajectory loss is calculated using the L1 norm, as shown in formula (2): wp i , Let i and j represent the i-th predicted trajectory point and the actual trajectory point, respectively.

7. The control prediction branch according to claim 3 is implemented in the following details: First, the visual features Fc of the front view and the bird's-eye view features Fb guided by high-level navigation commands are connected to generate control prediction features j. ctl This is used as the input feature of the control prediction module, as shown in formula (3): j ctl =MLP(Concat(F b ,F c )) (3) Then, guided by the trajectory points, a 128-dimensional trajectory feature is generated through a fully connected layer, which jointly controls the predicted feature j. ctl As shown, the input is a GRU network with a hidden layer dimension of 128 and a prediction step size of 4. This generates a multi-step control signal with a throttle and brake range of 0 to 1 and a steering wheel angle range of -1 to 1. This control signal serves as part of the input information for the subsequent control fusion module and is fused with the control signal output from the trajectory prediction module to generate the final control signal, which is then output to the vehicle's controller module. The control loss is calculated using the L1 norm, as shown in formula (4). Where ai, Let represent the actual control value and the predicted control value at time i, respectively.

8. The end-to-end motion planning method integrating a dynamic spatial attention mechanism according to claim 1, characterized in that, The loss function includes trajectory loss, control loss, and velocity loss. The velocity loss is defined as shown in formula (5): Where v i , Let αt represent the actual velocity value and the predicted velocity value at time i, respectively. The total loss is Loss = αt rj Loss trj +α ctl Loss ctl +α speed Loss peed , where α trj α ctl α speed These represent the weights of different losses.

Images