Automatic driving behavior planning method based on spatiotemporal characteristic perception and related device

Abstract

The application discloses a method for automatic driving behavior planning based on space-time characteristic perception and related devices, which comprises the following steps: determining a state space, an action space and a reward function of an automatic driving model according to an automatic driving simulator, constructing a partially observable Markov decision process (POMDP), establishing a deep neural network model for space-time characteristic perception, configuring training parameters for the deep neural network model for space-time characteristic perception, executing the automatic driving simulator and interacting with a deep reinforcement learning agent, and establishing a time-sequenced vehicle running state cache and extracting training data for the automatic driving simulation process. The application can mine potential space-time correlation information between vehicles from limited perception information, limit the calculation and analysis of invalid information, and thus realize more accurate and efficient control decisions generated by the system.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving behavior planning technology, and specifically relates to an autonomous driving behavior planning method and related devices based on spatiotemporal characteristic perception. Background Technology

[0002] With the rapid development of intelligent connected vehicles and artificial intelligence, autonomous driving technology is becoming increasingly common. Autonomous driving technology mainly consists of two aspects: environmental perception and control decision-making. The former utilizes different sensors to perceive the dynamic characteristics of driving, thereby providing an abstract representation of the driving environment; the latter generates actions such as steering, acceleration, and braking based on path and behavior planning strategies to control the vehicle. This invention focuses on behavior planning strategies for autonomous driving. Traditional behavior planning algorithms are implemented using rule-based or optimization-based methods, but these strategies are usually hand-designed and may contain potentially inaccurate assumptions, thus failing to scale to different real-world and complex scenarios, such as overtaking and roundabouts. Therefore, existing autonomous driving technologies focus more on using machine learning and deep learning methods to achieve autonomous driving. Among them, deep reinforcement learning, by combining the perception capabilities of deep neural networks with the decision-making capabilities of reinforcement learning, is widely used in the behavior planning of autonomous driving.

[0003] Autonomous driving based on deep reinforcement learning has attracted widespread attention both domestically and internationally. It utilizes different forms of environmental feature information (such as image data from onboard cameras, numerical data based on abstract states, and road condition information from road cameras) as input, and designs different autonomous driving algorithms based on different driving needs (efficiency, safety, comfort). However, due to sensor noise and obstructions, the perception ability of autonomous vehicles to perceive the driving environment is limited. How to extract potential feature information from this limited information is key to improving the performance of autonomous driving systems. Existing research on autonomous driving based on deep reinforcement learning mainly focuses on reward function design and state space fusion methods, but the limited perceptual information of the vehicle still restricts the performance of autonomous driving algorithms. Summary of the Invention

[0004] The purpose of this invention is to provide an autonomous driving behavior planning method and related devices based on spatiotemporal characteristic perception, in order to solve the problem that autonomous driving research based on deep reinforcement learning mainly focuses on reward function design, state space fusion and other methods, and the limited perception information of the vehicle restricts the performance of autonomous driving algorithms.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] Autonomous driving behavior planning methods based on spatiotemporal characteristic perception include:

[0007] Based on the autonomous driving simulator, the state space, action space and reward function of the autonomous driving model are determined, and a partially observable Markov decision POMDP is constructed.

[0008] A deep neural network model for spatiotemporal characteristic awareness is established based on partially observable Markov decision POMDP.

[0009] Configure training parameters for a deep neural network model for spatiotemporal characteristic perception, and execute an autonomous driving simulator to interact with a deep reinforcement learning agent;

[0010] Furthermore, the state space:

[0011] Vehicle driving characteristics are represented by position, speed, and heading. Assuming there are I-1 observable human-driven vehicles near the autonomous vehicle, the characteristics of the autonomous vehicle and the other observable human-driven vehicles are described as follows:

[0012] s t =(s i ) i∈[0,) where

[0013]

[0014] For autonomous vehicles, i = 0, x i and y i These refer to their longitudinal and transverse positions within the road, respectively. and These are the longitudinal and lateral velocities, respectively; cosψ i and sinψ i They are the headings ψ i The cosine and sine of the variable; the binary variable p represents whether it is observable near the autonomous vehicle.

[0015] Furthermore, the action space: The DRL agent is designed to take appropriate actions for action planning, including lane keeping, navigation, simple racing, overtaking, or maneuvering tasks; these tasks are accomplished through speed changes and steering control actions, using five high-level control decisions to construct the action space, including left turn, cruise, right turn, acceleration, and deceleration; high-level actions are translated into corresponding steering and throttle control signals to be manipulated by the low-level vehicle controllers to operate the autonomous vehicle.

[0016] Furthermore, the reward: The reward function is used to optimize the DRL policy and consists of the following three parts:

[0017] Collision assessment r c If an autonomous vehicle is involved in a collision, r c =-1, otherwise r c =0;

[0018] Stable speed evaluation r s :r c Used to reward speed limit [v] min v max [The faster speed v] t :

[0019]

[0020] Set the v according to the specific driving scenario. max and v min ;

[0021] Time progress assessment r h The time progress assessment of autonomous vehicles is defined as:

[0022]

[0023] Where, d h t is the distance between the front and rear of the vehicle. h This is the time progression threshold, set to 1.2 seconds; when d h Greater than t h v t Autonomous vehicles will be rewarded if they are rewarded, otherwise they will be penalized.

[0024] The above evaluation metrics are then combined into a single objective reward function, which is used to train the DRL policy at time step t:

[0025] r t =w c r c +w s r s +w h r h

[0026] Among them, w c w s and w h It is a positive weighted scalar.

[0027] Furthermore, a partially observable Markov decision POMDP is constructed:

[0028] POMDP is described as a quintuple. in These are the state space, action space, observation space, and reward; at each epoch t, ​​the DRL agent observes a state. It includes driving information from both the autonomous vehicle and surrounding human-driven vehicles, and then selects an action for the autonomous vehicle. As a control signal; after execution of a t Afterwards, the autonomous vehicle sends a reward to the agent. Then, the agent uses the transfer function. Get the next state s t+1 ;Right now in It is part of the observable space.

[0029] Furthermore, a deep neural network model for spatiotemporal characteristic perception is established:

[0030] Input part: Current state s t The action a at the previous moment t-1 Rewards r t-1 and the hidden feature state h generated by the recurrent neural network. t-1 ; where state s t It contains information about the speed, position, and heading of the autonomous vehicle itself and surrounding vehicles; action 'a' includes five types: acceleration, deceleration, left turn, right turn, and cruising; reward 'r' is used to guide the optimization of the deep reinforcement learning strategy; the feature state 'h' is initially set to all 0 and is then calculated by the recurrent neural network.

[0031] The time-domain awareness module consists of multiple linear layers and a gated recurrent unit (GRU) network; the linear layers are used to process the current state s. t The action a at the previous moment t-1 Rewards r t-1 and the hidden feature state h generated by the recurrent neural network. t-1 Feature extraction is performed; then, the extracted features are fused, and GRU is used to extract the current temporal feature information, and the hidden state is updated to h. t ;

[0032] The spatial awareness module consists of multiple linear layers and a vehicle-oriented multi-head attention layer; it stores the current state s. t After splitting the states according to different vehicles, different linear layers are used to encode these states. Then, the encoded states are concatenated with temporal feature information and decoded using linear layers. Finally, a query is generated using the decoded states of autonomous vehicles, and key-value pairs are constructed from the decoded states of all vehicles, thus mapping the query and key-value pairs to the network output.

[0033] Further, configure the training parameters:

[0034] Three autonomous driving scenarios are adopted: highway, two-lane overtaking, and roundabout. For the highway and two-lane overtaking driving scenarios, parameters such as road length, speed limit, number of lanes, initial vehicle speed, number of vehicles, and initial relative vehicle position are set. For the roundabout driving scenario, parameters such as the radius of the roundabout lane, lane entrances and exits in four directions, and vehicle spacing are set.

[0035] Establish a time-series vehicle operation status cache and training data extraction:

[0036] During each training iteration, the deep reinforcement learning agent selects a random action at each time step, executes the action, observes the feedback reward and the new state, and then stores the current time step state, action, reward and the next state into replay memory to build a temporal vehicle running state cache.

[0037] Training data is extracted from memory using randomized temporal batches. This means that a fixed batch size of time-series datasets is randomly extracted each time. Each time-series dataset represents a set of vehicle states from start to finish. Then, a starting point is randomly selected in each time-series dataset, and a fixed length of data is extracted to form a batch. Finally, the training is performed in temporal order.

[0038] Furthermore, an autonomous driving behavior planning system based on spatiotemporal characteristic perception includes:

[0039] The parameter determination module is used to determine the state space, action space, and reward function of the autonomous driving model based on the autonomous driving simulator, and to construct a partially observable Markov decision POMDP.

[0040] The model building module is used to build a deep neural network model that is aware of spatiotemporal characteristics based on partially observable Markov decision POMDP.

[0041] The training parameter configuration module is used to configure training parameters for a deep neural network model that perceives spatiotemporal characteristics, and to execute an autonomous driving simulator that interacts with a deep reinforcement learning agent.

[0042] Furthermore, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of an autonomous driving behavior planning method based on spatiotemporal characteristic perception.

[0043] Furthermore, a computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of an autonomous driving behavior planning method based on spatiotemporal characteristic perception.

[0044] Compared with the prior art, the present invention has the following technical effects:

[0045] This invention designs a neural network that integrates a recurrent neural network (RNN) and an attention mechanism. This neural network model is implemented by fusing RNN and attention techniques. It uses the current state, the previous action, the reward, and the hidden feature state generated by the RNN as input. The RNN further updates the hidden features in the temporal domain, and an attention mechanism based on vehicle state is used to integrate the current state and the hidden feature state as input. This allows the system to perceive the spatial dependencies between vehicles, extracting potential spatiotemporal correlations between vehicles from limited perceptual information and limiting the computation and analysis of invalid information. This enables the system to generate more accurate and efficient control decisions. The method proposed in this invention has strong scalability and can be applied to various autonomous driving scenarios. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the model of the present invention.

[0047] Figure 2 Performance evaluation of different training strategies using different algorithms. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings:

[0049] Please see Figure 1 This model can utilize GRU to extract more effective temporal hidden information and employ an attention mechanism to extract the temporal and spatial dependencies between the autonomous vehicle itself and surrounding vehicles, thereby generating a more effective and safer control strategy and ensuring the performance of the autonomous driving system. The implementation of this invention's model is simulated in three different autonomous driving scenarios: highways, two-lane overtaking, and roundabouts, ensuring the efficiency and universality of the method.

[0050] The specific implementation of this invention includes the following steps:

[0051] Step 1: Based on the autonomous driving simulator, determine the state space, action space, and reward function of the autonomous driving model;

[0052] Lane change decisions and speed control tasks for autonomous vehicles can be formulated as Markov Decision Processes (MDPs). However, due to sensor noise and obstructions, autonomous vehicles can only perceive a portion of their surroundings. Therefore, a Partially Observable Markov Decision Process (POMDP) ​​needs to be constructed for autonomous driving.

[0053] Formally, a POMDP can be described as a quintuple. in These are the state space, action space, observation space, and reward. At each epoch t, ​​the DRL agent observes a state. It includes driving information from both the autonomous vehicle and surrounding human-driven vehicles, and then selects an action for the autonomous vehicle. (i.e., control signal). After executing a... t Afterwards, the autonomous vehicle sends a reward to the agent. Then, the agent uses the transfer function. Get the next state s t+1 Due to partial observability, the agent can only observe a portion of the state, i.e. in It is part of the observable space.

[0054] State space: Vehicle driving characteristics can be represented by position, speed, and heading. Assuming there are I-1 observable human-driven vehicles near the autonomous vehicle, this invention describes the characteristics of the autonomous vehicle and the other observable human-driven vehicles as follows:

[0055] s t =(s i ) i∈[0，I) ,where (1)

[0056]

[0057] For autonomous vehicles (i.e., i = 0), x i and y i These refer to their longitudinal and transverse positions within the road, respectively. and These are the longitudinal and lateral velocities, respectively; cosψ i and sinψ i They are the headings ψ i The cosine and sine of the value. For other manually driven vehicles (i.e., i > 0), these position variables (x) i and y i ) and velocity variables ( and Related to autonomous vehicles, for example The binary variable p represents whether the autonomous vehicle is observable in the vicinity of the vehicle.

[0058] Action Space: The DRL agent is designed to take appropriate actions for action planning, including lane keeping, navigation, simple racing, overtaking, or maneuvering tasks. These tasks can be accomplished through longitudinal (speed changes) and lateral (steering control) actions. Therefore, this invention uses five high-level control decisions to construct the action space, including left turn, cruise, right turn, acceleration, and deceleration. These high-level actions are translated into corresponding steering and throttle control signals for the lower-level vehicle controller to operate the autonomous vehicle.

[0059] Reward: The reward function is used to optimize the DRL policy and consists of the following three parts:

[0060] Collision assessment r c If an autonomous vehicle is involved in a collision, r c =-1, otherwise r c It is 0.

[0061] Stable speed evaluation r s :r c Used to reward speed limit [v] min v max [The faster speed v] t :

[0062]

[0063] Speed ​​limits may vary on different roads, so you need to set the speed limit based on the specific driving scenario. max and v min .

[0064] Time progress assessment r h To further prioritize safety, this invention defines the time progress assessment of autonomous vehicles as follows:

[0065]

[0066] Where, d h t is the distance between the front and rear of the vehicle. h This is the time progression threshold, set to 1.2 seconds. When d h Greater than t h v t Autonomous vehicles will be rewarded if they are rewarded, otherwise they will be punished.

[0067] Given these evaluation metrics, they are then combined into a single objective reward function for training the DRL policy at time step t:

[0068] r t =w c r c +w s r s +wh r h (4)

[0069] Among them, w c w s and w h It is a positive weighted scalar.

[0070] Step 2: Establish a deep neural network model for spatiotemporal characteristic awareness and configure the neural network size;

[0071] Given a POMDP, the goal of a DRL agent is to maximize the expected value of the long-run reward. Starting from the initial state s... i Initially, the agent will learn a policy π(a|s) that maps each state s to an action a, where the chosen action a maximizes s. i The expected reward for all subsequent steps. This policy can be evaluated using its action-value function, also known as the Q-function:

[0072]

[0073] Where γ∈[0,1] is the discount factor, used to specify the importance between future rewards and current rewards; that is, γ=0 indicates that the agent only focuses on immediate rewards, while γ=1 indicates that the agent seeks long-term rewards. Using the Bellman equation and time difference, the Q function can be simplified to:

[0074]

[0075] Where γ is the learning rate, and a′ maximizes all possible Q(s′, a′). The previous Q is taken... t-1 (s, a), and add the time difference factor γ to obtain the new Q. t (s, a).

[0076] Using the Q-function, the policy π can be learned through the DQN algorithm. However, since the autonomous driving environment is partially observable, finding the optimal policy for POMDP is very difficult. Integrating recurrent neural networks (RNNs) to find hidden states over time is an effective solution to the POMDP problem. Traffic models and their changing patterns have high spatiotemporal correlation; therefore, this invention designs a spatiotemporal DNN for learning DRL policies based on RNNs and attention mechanisms, and calls this DRL algorithm DAGQN.

[0077] First, the dynamic changes in historical driving patterns of different vehicles and future traffic conditions over time are crucial for making rational action decisions. This invention uses an RNN to aggregate observation history over time and identify these temporal features to generate an optimal policy. Feeding more operational information (e.g., observations, actions, and rewards) into the RNN can produce policies with better performance; therefore, the input to DAGQN includes the currently observed states. t The previously performed action a t-1 and the reward r t-1 This invention will be derived from s t a t-1 and r t-1 The extracted features are concatenated into a one-dimensional tensor f c Furthermore, further from the previously hidden state h t-1 The extracted features are added to f c In the middle, more historical information is captured. Finally, a gated recurrent unit (GRU) layer is used to capture temporal information. The calculation process for this part is as follows:

[0078]

[0079]

[0080]

[0081] Where <> indicates series connection; It is a fully connected network with n linear layers, where each layer is an affine transformation with a weight matrix W and a bias b. The bias is set to 0, that is... Apart from All linear layers are followed by a ReLU activation function. The temporal characteristic f g The output consists of a linear layer and a ReLU activation function.

[0082] Secondly, the spatial location and interactions of vehicles provide another source of information for behavior planning. Filtering relevant spatial information from spatial features and capturing the vehicle's dependencies are essential for the agent to make effective decisions. Therefore, DAGQN uses a multi-head social attention mechanism to force the agent to pay more attention to vehicles that influence the autonomous vehicle's planned route. In the attention network, this invention first... t Break it down into separate states for each autonomous vehicle and each manually driven vehicle. Then through the linear layer (for autonomous vehicles) Other manually driven vehicles are Extract features from it, and compare the features with the extracted time features f′. g Connect them.

[0083] Then, through the linear layer and Encode these individual features into a size d x The embedded space is used to feed these elements into the self-attention layer. In the self-attention layer, the autonomous vehicle uses linear projection... Send a single All vehicles are projected linearly. and Send a set of keys respectively and a value collection In each header, the query is processed via a dot product. With each key k i Compare, by Scaling is performed, normalization is achieved using the softmax function σ, and finally, the result is used to converge the value v. i The outputs of all heads are combined with a linear layer to obtain the final Q-value. The calculation process of the attention mechanism is as follows:

[0084]

[0085] Additionally, for DRL models, the following network size settings are available:

[0086] Step 3: Configure training parameters and run the autonomous driving simulator to interact with the deep reinforcement learning agent.

[0087] This invention employs three typical autonomous driving scenarios: highway, two-lane overtaking, and roundabout. In the highway scenario, there are four lanes in one direction, the road length is 10,000 meters, and the permissible vehicle speed range is 20 m / s to 30 m / s. The initial speed v0 of all vehicles is randomly selected between 21 m / s and 27 m / s. Forty manually driven vehicles and one autonomous vehicle are randomly generated, each with an initial lane ID (l ID Let be any value in [0, 3], and the initial longitudinal distance between vehicles follows a uniform distribution. Where n lane This refers to the number of lanes. In a two-lane overtaking scenario, the road length is 1500 meters, and the speed limit is 16 m / s. The forward lane (l ID =0) and the opposite lane (l ID =1) Ten manually driven cars are randomly generated, with initial speeds in the ranges of [6, 10] m / s and [4, 12] m / s, respectively. The initial longitudinal distance between the cars follows a distribution. sp 0=5, sp 1 =7. Considering the driving conditions of manually driven vehicles in the opposite lane, autonomous vehicles in the forward lane need to overtake the vehicles in front. In a roundabout scenario, there are two intersections with radii of 60 meters (l ID =0) and 64 meters (l ID =1) A circular driveway. There is one entrance / exit lane and one lane in each of the four cardinal directions (east, west, north, and south). ID =1 connected. At each entrance / exit lane, 3 manually driven vehicles are randomly generated, with a distance between the vehicles in the range of [10, 60] meters. On the circular lane, 6 manually driven vehicles with a spacing parameter sp = 2 are randomly generated, with initial speeds in the entrance and circular lanes ranging from [9, 15] to [7, 13] m / s, respectively.

[0088] This invention uses the default parameters of IDM and MOBIL defined in the highway-env simulator to implement longitudinal and lateral control of manually driven vehicles. The control sampling frequency of the manually driven vehicles is 5Hz, meaning an action is taken once every 0.2 seconds. Simultaneously, the autonomous vehicle is configured to observe the six manually driven vehicles ahead, i.e., I = 7. For model training, the replay memory D has a capacity M = 60000, the batch size is 4, the sequence length of each batch is less than 30, the discount factor γ = 0.9, the network is updated every C = 50 time steps, and the learning rate is 0.001. The coefficient w of the reward function... c w s w h The values ​​were set to 1.2, 0.4, and 0.4 respectively. Furthermore, during network training, ∈ was set to decrease from 0.9 by 0.002 each time.

[0089] Furthermore, during each training iteration, the deep reinforcement learning agent selects a random action at each time step, executes the action, observes the feedback reward and the new state, and then stores the current time step's state, action, reward, and next state in replay memory to establish a vehicle operation state cache. To update the network, training data needs to be extracted from the replay memory. Traditional training methods use completely random batch extraction, which destroys the original sequence features. This invention, to ensure effective perception of the temporal features of the vehicle state, employs a temporally sequenced vehicle operation state cache and training data extraction method. Specifically, a fixed-batch size temporal dataset is randomly extracted each time, each representing a set of vehicle states from start to end. Then, a starting point is randomly selected within each temporal dataset, and a fixed length of data is extracted to form a batch. Finally, the data is trained sequentially during training. To evaluate the performance of the DAGQN algorithm in different driving scenarios, it is compared with DQN, DRQN, and DQN with a self-attention layer (EADQN). The results are as follows: Figure 2 As shown. Figure 2 In the diagram, H, T, and R represent highway, two-lane, and roundabout scenarios, respectively. Evaluation metrics CR, AS, ER, RD, and LDD represent the collision rate, average speed, average round reward, running time, and longitudinal travel distance obtained by the AV over 40 iterations, respectively. It can be seen that the DAGQN algorithm exhibits better performance in different driving scenarios. Furthermore, theoretically, LDD should equal AS*RD, but in reality, due to lateral changes and collisions, they are not equal. For example, in the highway scenario, due to multiple lateral changes, the actual LDD of the AV is lower than AS*RD, because AS consists of longitudinal and lateral speeds. In the two-lane scenario, the LDD of DRQN ​​is significantly smaller than AS*RD because the collision rate of the DRQN-controlled AV when overtaking is high, and collisions in the opposite lane cause it to move backward. The strategies of DQN and EADQN rarely allow the AV to overtake, therefore their AS is lower, but their RD is higher, and their LDD is closer to AS*RD.

[0090] In summary, DRQN ​​can capture temporal driving features through RNN, while EADQN can capture vehicle-to-ego features through attention mechanisms. Therefore, DRQN ​​and EADQN can achieve better performance than DQN. Since the DAGQN of this invention can capture both spatial and temporal features, the AV controlled by the DAGQN strategy outperforms other vehicles in different driving scenarios.

[0091] In another embodiment of the present invention, a method is provided. Autonomous driving behavior planning system based on spatiotemporal characteristics perception able to use To achieve the above Autonomous driving behavior planning based on spatiotemporal characteristics perception Specifically, the system includes the following methods:

[0092] The parameter determination module is used to determine the state space, action space, and reward function of the autonomous driving model based on the autonomous driving simulator, and to construct a partially observable Markov decision POMDP.

[0093] The model building module is used to build a deep neural network model that is aware of spatiotemporal characteristics based on partially observable Markov decision POMDP.

[0094] The training parameter configuration module is used to configure training parameters for a deep neural network model that perceives spatiotemporal characteristics, and to execute an autonomous driving simulator that interacts with a deep reinforcement learning agent.

[0095] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0096] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve corresponding method flows or corresponding functions. The processor described in this embodiment of the present invention can be used for the operation of an autonomous driving behavior planning method based on spatiotemporal characteristic perception.

[0097] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the autonomous driving behavior planning method based on spatiotemporal characteristic perception in the above embodiments.

[0098] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for autonomous driving behavior planning based on spatiotemporal characteristics awareness, characterized in that, Comprise: According to the automatic driving simulator, determine the state space, action space and reward function of the automatic driving model, and build a partially observable Markov decision POMDP; Based on the partially observable Markov decision POMDP, a deep neural network model for spatiotemporal characteristic perception is established; For the deep neural network model for spatiotemporal characteristic perception, configure the training parameters, execute the automatic driving simulator, and interact with the deep reinforcement learning agent; Build a partially observable Markov decision POMDP: A POMDP is described as a five-tuple where , , , are the state space, action space, observation space and reward, respectively; at each time period , a DRL agent observes a state which includes the driving information of the autonomous vehicle and the surrounding human-driven vehicles, and then selects an action as the control signal for the autonomous vehicle; after the execution , the autonomous vehicle feeds back a reward to the agent; After that, the agent obtains the next state according to the transition function ; that is where is the partially observable space; Establish a deep neural network model for spatiotemporal characteristic perception: input: current state , action at previous time step , reward , and hidden feature state generated by recurrent neural network ; where state contains information about speed, position and heading of the ego vehicle and its surrounding vehicles; action contains five types of actions: accelerate, decelerate, left turn, right turn and cruise; reward is used to guide the optimization of deep reinforcement learning policy; Feature state Initial values are all 0, and then calculated by recurrent neural network; The time domain perception module is composed of multiple linear layers and a gated recurrent unit network GRU; the linear layers are respectively used for feature extraction on the current state , the action of the last moment , the reward , and the hidden feature state generated by the recurrent neural network ; then the extracted features are fused, and the current time domain feature information is extracted by using the GRU, and the hidden state is updated to ; Space-aware module: composed of multiple linear layers and a vehicle-oriented multi-head attention layer; the current state After splitting according to different vehicles, different linear layers are used to encode these states; then the encoded states are respectively connected with the time domain feature information, and then decoded using linear layers; then, a query is generated using the decoded state of the autonomous vehicle, and the decoded states of all vehicles are constructed into key-value pairs, so as to map the query and the key-value pairs into the output of the network. 2.The spatio-temporal characteristic-aware autonomous driving behavior planning method of claim 1, wherein, State space: The vehicle motion characteristics are represented in terms of position, velocity, and heading, assuming that there is one observable human-driven vehicle in the vicinity of the autonomous vehicle The characteristics of the autonomous vehicle and the other observable human-driven vehicles are described as follows: , For an autonomous vehicle , and are the longitudinal and lateral position in the road, respectively; and are the longitudinal and lateral speed, respectively; and are the cosine and sine of the heading , respectively; binary variable indicates whether an object is observable in the vicinity of the autonomous vehicle. 3.The spatio-temporal characteristic-aware autonomous driving behavior planning method of claim 1, wherein, Action space: DRL agent aims to take appropriate actions for action planning, including lane keeping, navigation, simple racing, overtaking or maneuvering tasks; these tasks are completed through speed change and steering control actions, using five high-level control decisions to build the action space, including left turn, cruise, right turn, acceleration and deceleration; High-level actions are converted into corresponding steering and throttle control signals to control the autonomous vehicle by low-level vehicle controllers. 4.The spatio-temporal characteristic-aware autonomous driving behavior planning method of claim 1, wherein, Reward: The reward function is used to optimize the DRL policy, which consists of the following three parts: Collision assessment : If the autonomous vehicle is in a collision, -1, otherwise 0; Stable speed assessment : For rewarding faster speed within speed limit :​ Setting according to a specific driving scenario and ; Time Progress Assessment : Time Progress Assessment of an autonomous vehicle is defined as: wherein, is the distance between the vehicle head; is a time progress threshold, set to 1.2s; when is greater than the autonomous vehicle will be rewarded, otherwise it will be penalized; The above evaluation metrics are then combined into a single target reward function, used to update the DRL policy at time step Training the DRL policy: wherein , and are positive weighting scalars. 5.The spatio-temporal characteristic-aware autonomous driving behavior planning method of claim 1, wherein, Configure the training parameters: Three automatic driving scenarios are adopted: highway, double-lane overtaking and roundabout intersection, and the road length, speed limit, lane number, vehicle initial speed, vehicle number and vehicle initial relative position parameters are set for highway and double-lane overtaking driving scenarios; The roundabout intersection driving scenario sets the roundabout lane radius, four-direction lane entrance and exit and vehicle spacing parameters; Establish a time-sequential vehicle running state cache and training data extraction: During each training iteration, the deep reinforcement learning agent selects a random action at each time step, executes the action and observes the feedback reward and new state, then stores the state, action, reward and next state of the current time step into the replay memory to establish a time-sequential vehicle running state cache; Extract training data from the memory, use a random time-sequential batch, that is, randomly extract a fixed batch size of time-sequential data sets each time, each time-sequential data set represents a set of states from the beginning to the end of the vehicle, then in each time-sequential data set, a starting point is randomly selected, and a fixed length of data is extracted to form a batch, and finally the batch is trained in time sequence during training.

6. An autonomous driving behavior planning system based on spatiotemporal characteristics awareness, characterized in that, Comprise: Parameter determination module, for determining the state space, action space and reward function of the automatic driving model according to the automatic driving simulator, and building a partially observable Markov decision POMDP; Model establishment module, for establishing a deep neural network model for spatiotemporal characteristic perception based on the partially observable Markov decision POMDP; Training parameter configuration module, for configuring training parameters for the deep neural network model for spatiotemporal characteristic perception, and executing the automatic driving simulator and interacting with the deep reinforcement learning agent; Build a partially observable Markov decision POMDP: A POMDP is described as a five-tuple , where , , , , , , are the state space, action space, observation space and reward, respectively; at each time period , a DRL agent observes a state , which includes the driving information of the autonomous vehicle and the surrounding human-driven vehicles, and then selects an action as the control signal for the autonomous vehicle; after the execution , the autonomous vehicle feeds back a reward to the agent; After that, the agent obtains the next state according to the transition function ; that is where is the partially observable space; Establish a deep neural network model for spatiotemporal characteristic perception: input: current state , previous action , reward , and hidden feature state generated by recurrent neural network ; wherein state contains information about speed, position and heading of the ego vehicle and its surrounding vehicles; action contains five types of actions: accelerate, decelerate, left turn, right turn and cruise; reward is used to guide the optimization of deep reinforcement learning policy; Feature state Initial values are all 0, and are then calculated by a recurrent neural network; The time domain perception module is composed of multiple linear layers and a gated recurrent unit network GRU; the linear layers are respectively used for feature extraction on the current state , the action of the last moment , the reward , and the hidden feature state generated by the recurrent neural network ; then the extracted features are fused, the current time domain feature information is extracted by using the GRU, and the hidden state is updated to ; Space-aware module: composed of multiple linear layers and a vehicle-oriented multi-head attention layer; encodes the current state After splitting according to different vehicles, different linear layers are used to encode these states; then the encoded states are respectively connected with the time domain feature information, and then decoded using linear layers; then, a query is generated using the decoded state of the autonomous vehicle, and the decoded states of all vehicles are constructed into key-value pairs, so as to map the query and the key-value pairs into the output of the network.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the spatiotemporal characteristic perception-based automatic driving behavior planning method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program, when executed by a processor, implements the steps of the method for planning an autonomous driving behavior based on spatiotemporal characteristic perception according to any one of claims 1 to 5.

Images