Safe driving decision-making method for large commercial vehicles in urban low-speed environment

Abstract

The present invention discloses a safe driving decision-making method for large commercial vehicles in a low-speed urban environment. First, the safe driving behavior of human drivers in an urban traffic environment is collected to construct a safe driving behavior data set. Secondly, a safe driving decision-making model for commercial vehicles based on multi-head attention is constructed. The model includes two sub-networks: a deep double Q network and a generative adversarial imitation learning sub-network. Among them, the deep double Q network learns safe driving strategies in edge scenarios such as dangerous scenarios and conflict scenarios through unsupervised learning; the generative adversarial imitation learning sub-network imitates the safe driving behavior of human drivers under different driving conditions and driving conditions. Finally, the safe driving decision-making model is trained to obtain driving strategies under different driving conditions and driving conditions. The method proposed in the present invention can simulate the safe driving behavior of human drivers, and takes into account the impact of factors such as visual blind spots and sudden obstacles on driving safety.

Description

Technical Field

[0001] The invention relates to a driving decision method for an operating vehicle, and in particular to a safe driving decision method for a large operating vehicle in a low-speed urban environment, and belongs to the technical field of automobile safety. Background Art

[0002] In urban traffic environments, road traffic accidents caused by driver blind spots account for the highest proportion, and the main causes of these accidents are large commercial vehicles such as heavy trucks, large buses, and car trains. Unlike passenger vehicles, large commercial vehicles have the characteristics of large size, long body, large wheelbase, and high driving position. There are many static and dynamic visual blind spots around their bodies, such as in front of the front of the vehicle, near the right front wheel, and below the right rearview mirror. When commercial vehicles turn, especially when turning right, they are very likely to collide with or even run over pedestrians and non-motor vehicles in the blind spot of vision, which is the main area for serious safety accidents. In addition, compared with the relatively closed highway scene, in the urban traffic environment where motor vehicles and non-motor vehicles are mixed, the types and numbers of traffic participants are relatively large, and commercial vehicles often encounter obstacles, which is more dangerous. Therefore, in an open urban traffic environment with multiple traffic targets, how to improve the driving safety of commercial vehicles is a key issue that needs to be solved urgently, and it is also the focus of ensuring urban road traffic safety.

[0003] At present, the active development of autonomous driving technology has become an important means of ensuring vehicle operation safety that is widely recognized at home and abroad. As a key link in achieving high-quality autonomous driving, driving decisions determine the rationality and safety of autonomous driving of commercial vehicles. If the driver can be given a danger warning 1.5 seconds before a traffic accident occurs and a reliable and effective safe driving strategy is provided, the frequency of traffic accidents caused by factors such as visual blind spots and sudden obstacles can be greatly reduced. Therefore, studying the safe driving decision-making method of large commercial vehicles plays an important role in ensuring the driving safety of commercial vehicles.

[0004] Many patents and literature have studied the anti-collision driving decision-making, but they are mainly for passenger vehicles. Compared with passenger vehicles, commercial vehicles have larger visual blind spots and longer braking distances and braking times. The anti-collision decision-making method for passenger vehicles cannot be directly applied to commercial vehicles. On the other hand, some patents have studied the safe driving decision-making of commercial vehicles, such as a highly human-like safe driving decision-making method for autonomous driving commercial vehicles (application number: 202210158758.2), a lane change decision method for large commercial vehicles based on deep learning (publication number: CN113954837A), etc., but these decision-making methods are all aimed at highway scenarios.

[0005] Unlike highway scenarios with fewer types of traffic participants, urban traffic environments are open, with multiple traffic target interference, and mixed traffic of vehicles and non-vehicles. In particular, the presence of factors such as vehicle visual blind spots and sudden obstacles poses a higher challenge to the safe driving of commercial vehicles in urban traffic environments. Therefore, the safe driving decision-making method for commercial vehicles in highway scenarios cannot be directly applied to open and disturbed urban traffic environments.

[0006] In general, in an open urban traffic environment with multiple traffic interferences, the existing methods are difficult to meet the requirements of commercial vehicles for safe driving decisions. There is still a lack of safe driving decision-making methods that can provide specific driving suggestions such as driving actions and driving paths, especially the lack of research on safe driving decisions for large commercial vehicles that considers the impact of visual blind spots and sudden obstacles. Summary of the invention

[0007] Purpose of the invention: In order to achieve safe driving decisions for large commercial vehicles in low-speed urban environments and ensure vehicle driving safety, the present invention proposes a safe driving decision method for large commercial vehicles in low-speed urban environments for automatic driving commercial vehicles such as heavy trucks and heavy trucks. This method comprehensively considers the impact of factors such as visual blind spots, sudden obstacles, and different driving conditions on driving safety, and can simulate the safe driving behavior of human drivers to provide more reasonable and safe driving strategies for automatic driving commercial vehicles, which can effectively ensure the driving safety of automatic driving commercial vehicles. At the same time, this method does not need to consider complex vehicle dynamics equations and body parameters, the calculation method is simple and clear, and the safe driving strategy of automatic driving commercial vehicles can be output in real time. The sensors used are low in cost, which is convenient for large-scale promotion.

[0008] Technical solution: To achieve the purpose of the present invention, the technical solution adopted by the present invention is: a safe driving decision-making method for large commercial vehicles in a low-speed urban environment. First, the safe driving behavior of human drivers in an urban traffic environment is collected to construct a safe driving behavior data set. Secondly, a safe driving decision-making model for commercial vehicles based on multi-head attention is constructed. The model contains two sub-networks: a deep dual-Q network and a generative adversarial imitation learning sub-network. Among them, the deep dual-Q network learns safe driving strategies in edge scenarios such as dangerous scenarios and conflict scenarios through unsupervised learning; the generative adversarial imitation learning sub-network imitates safe driving behaviors under different driving conditions and driving conditions. Finally, the safe driving decision-making model is trained to obtain driving strategies under different driving conditions and driving conditions, and to achieve high-level decision output for the safe driving behavior of commercial vehicles. Specifically, the following steps are included:

[0009] Step 1: Collect safe driving behaviors of human drivers in urban traffic environments

[0010] In order to achieve driving decisions comparable to those of human drivers, the present invention collects safe driving behaviors under different driving conditions and driving conditions through actual road tests and driving simulations, and then constructs a data set that characterizes the safe driving behaviors of human drivers. Specifically, it includes the following 4 sub-steps:

[0011] Sub-step 1: Use millimeter-wave radar, 128-line lidar, visual sensor, Beidou sensor and inertial sensor to build a multi-dimensional target information synchronous acquisition system.

[0012] Sub-step 2: In a real urban environment, multiple drivers drive an operating vehicle equipped with a multi-dimensional target information synchronous acquisition system in turn, and collect and process relevant data on the driver's lane changing, lane keeping, vehicle following, acceleration and deceleration, etc., to obtain multi-source heterogeneous description data of each driving behavior, such as obstacle distances in multiple directions measured by radar or visual sensors, position, speed, acceleration and yaw angular velocity measured by Beidou sensors and inertial sensors, and steering wheel angle measured by vehicle-mounted sensors.

[0013] Sub-step 3: In order to simulate safe driving behaviors in edge scenarios such as dangerous scenarios and conflict scenarios, a virtual city scenario based on hardware-in-the-loop simulation is built. The constructed urban traffic scenarios include the following three categories:

[0014] (1) When the vehicle is driving, there is a traffic participant approaching laterally in front of the vehicle (i.e., encountering an obstacle suddenly);

[0015] (2) When the vehicle is turning, there are stationary traffic participants in the vehicle's blind spot;

[0016] (3) When the vehicle is turning, there are moving traffic participants in the vehicle’s blind spot.

[0017] In the above traffic scenarios, there are various road network structures (straight roads, curves and intersections) and multiple types of traffic participants (commercial vehicles, passenger cars, non-motor vehicles and pedestrians).

[0018] Multiple drivers drive the operating vehicles in the virtual scene using real controllers (steering wheel, accelerator and brake pedal) to collect information such as the vehicle's lateral and longitudinal position, lateral and longitudinal speed, lateral and longitudinal acceleration, relative distance and relative speed to surrounding traffic participants.

[0019] Sub-step 4: Based on the data collected from the real urban environment and the driving simulation environment, a driving behavior dataset for safe driving decision learning is constructed, which can be specifically expressed as:

[0020]

[0021] In the formula, X represents a binary group covering states and actions, that is, a dataset constructed to characterize the safe driving behavior of human drivers, (s j ,a j ) represents the state-action pair at time j, where s j represents the state at time j, a j represents the action at time j, i.e. the human driver’s action based on state s j The action taken, n is the number of "state-action" pairs in the database.

[0022] Step 2: Construct a safe driving decision model for commercial vehicles based on multi-head attention

[0023] In order to achieve safe driving decisions for large commercial vehicles in low-speed urban environments, the present invention comprehensively considers the impact of factors such as visual blind spots, sudden obstacles, and driving conditions on driving safety, and establishes a safe driving decision-making model for commercial vehicles. Considering that deep reinforcement learning combines the perception ability of deep learning with the decision-making ability of reinforcement learning, and explores the traffic environment through unsupervised learning, the present invention uses deep reinforcement learning to learn safe driving strategies in edge scenarios such as dangerous scenarios and conflict scenarios. In addition, considering that imitation learning has the ability to emulate role models, the present invention uses imitation learning to simulate the safe driving behavior of human drivers under different driving conditions and driving conditions. Therefore, the constructed safe driving decision model consists of two parts, which are specifically described as follows:

[0024] Sub-step 1: Define the basic parameters of the safe driving decision model

[0025] Firstly, the safe driving decision-making problem in a low-speed urban environment is transformed into a finite Markov decision process. Secondly, the basic parameters of the safe driving decision-making model are defined.

[0026] (1) Define the state space

[0027] In order to describe the motion state of the vehicle and nearby traffic participants, the present invention uses time series data and occupancy grid graph to construct the state space. The specific description is as follows:

[0028] S t =[S1(t),S2(t),S3(t)] (2)

[0029] In the formula, S t represents the state space at time t, S1(t) and S2(t) represent the state space associated with the time series data at time t, and S3(t) represents the state space associated with the occupancy grid at time t.

[0030] First, the motion state of the vehicle is described using continuous position, velocity, acceleration, and heading information:

[0031] S1(t)=[p x ,p y ,v x ,v y ,a x ,a y ,θ s ] (3)

[0032] In the formula, p x ,p y Respectively represent the lateral position and longitudinal position of the vehicle, in meters, v x ,v y They represent the lateral speed and longitudinal speed of the vehicle, in meters per second, and a x ,a y They represent the lateral acceleration and longitudinal acceleration of the vehicle, respectively, in meters per second squared, and θ s Indicates the heading angle of the vehicle in degrees.

[0033] Secondly, the relative motion state information between the vehicle and surrounding traffic participants is used to describe the motion state of surrounding traffic participants:

[0034]

[0035] In the formula, They represent the relative distance, relative speed and acceleration between the ego-vehicle and the i-th traffic participant, in meters, meters per second and meters per second squared, respectively.

[0036] In the existing state space definition methods, fixed coding methods are often used, that is, the number of surrounding traffic participants considered is fixed. However, in actual urban traffic scenarios, the number and positions of traffic participants around operating vehicles are constantly changing, and side collisions caused by sudden obstacles and visual blind spots need to be specially considered. Although the fixed coding method can achieve effective state representation, the number of traffic participants considered is limited (the minimum amount of information required to represent the scene is used), and it is impossible to accurately and comprehensively describe the impact of all surrounding traffic participants on the driving safety of operating vehicles.

[0037] Finally, in order to more vividly describe the relative position relationship between the vehicle and surrounding traffic participants and improve the reliability and effectiveness of decision-making, the present invention grids the road area into several a×b grid areas, and abstracts the road area and vehicle targets into a grid map, that is, the "existence" grid map S3(t) used to describe the relative position relationship. Where a represents the length of the grid area, and b represents the width of the grid area.

[0038] The "existence" grid map contains four attributes, including grid coordinates, whether there is a vehicle, the type of the corresponding vehicle, and the distance from the left and right lane lines. Among them, the grid without traffic participants is set to "0", and the grid with traffic participants is set to "1". The position distribution of this grid and the grid where the vehicle is located is used to describe the relative distance between the two vehicles.

[0039] (2) Define the action space

[0040] Define the action space using lateral and longitudinal driving actions:

[0041] A t =[a left ,a straight ,a right ,a accel ,a cons ,a decel ] (5)

[0042] In the formula, A t represents the action space at time t, a left ,a straight ,a right They represent left turn, straight ahead and right turn respectively. accel ,a cons ,a decel They represent acceleration, constant speed, and deceleration respectively.

[0043] (3) Define the reward function

[0044] R t =r1+r2+r3 (6)

[0045] In the formula, R t represents the reward function at time t, r1, r2, r3 represent the forward collision avoidance reward function, the backward collision avoidance reward function, and the side collision avoidance reward function, respectively, which can be obtained by equations (7), (8), and (9).

[0046]

[0047]

[0048]

[0049] Where, TTC represents the time when the ego vehicle collides with the obstacle in front, which can be obtained by dividing the distance between the ego vehicle and the obstacle in front by the relative speed. thr Indicates the distance to collision time threshold, RTTC indicates the rearward collision time, RTTC thr Represents the rear collision time threshold, in seconds, x lat represents the distance between the vehicle and the traffic participants on both sides, xmin represents the minimum lateral safety distance in meters. β1, β2, and β3 represent the weight coefficients of the forward collision avoidance reward function, the backward collision avoidance reward function, and the lateral collision avoidance reward function, respectively.

[0050] Sub-step 2: Construct a decision sub-network based on a deep double Q network

[0051] Considering that the Double Deep Q Network (DDQN) improves the efficiency of data utilization by using the experience reuse pool, and can avoid parameter oscillation or divergence, it can reduce the negative learning effect caused by overestimation in the Q learning network. Therefore, the present invention uses the deep double Q network to learn safe driving strategies in edge scenarios.

[0052] Unlike processing a state space with fixed dimensions, processing feature information covering all surrounding traffic participants requires stronger feature extraction capabilities. Considering that the attention mechanism can capture richer feature information (the dependency between the vehicle and surrounding traffic participants), the present invention designs a policy network based on a multi-head attention mechanism. In addition, considering that driving decisions are only related to the motion state of the vehicle and surrounding traffic participants, and should not be affected by the order of traffic participants in the state space, the present invention uses a position encoding method (literature: Vaswani, Ashish, et al. "Attention is all you need." Advances in Neural Information Processing Systems. 2017.) to build permutation invariance into the decision subnetwork.

[0053] The attention layer can be expressed as:

[0054] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (10)

[0055] In the formula, MultiHead(Q,K,V) represents the multi-head attention value, Q represents the query vector, K represents the key vector, and the dimension is d k , V represents the value vector, dimension is d v , W O Represents the parameter matrix that needs to be learned, head h represents the hth head in the multi-head attention. In the present invention, h=2, which can be calculated by the following formula:

[0056]

[0057]

[0058] In the formula, Attention(Q,K,V) represents the output attention matrix, Represents the parameter matrix that needs to be learned.

[0059] A decision subnetwork based on a deep double-Q network is constructed, as described below.

[0060] First, the state space S t They are connected to encoder 1, encoder 2 and encoder 3 respectively. Encoder 1 consists of two fully connected layers and outputs the vehicle motion state encoding. The structure of encoder 2 is the same as encoder 1 and outputs the relative motion state encoding. Encoder 3 consists of two convolutional layers and outputs the occupation grid map encoding.

[0061] The number of neurons in the fully connected layer is 64, the activation function is the Tanh function, the convolution kernel of the convolution layer is 3×3, and the step size is 2.

[0062] Secondly, the multi-head attention mechanism is used to analyze the dependency between the ego vehicle and surrounding traffic participants, so that the decision sub-network can notice traffic participants that suddenly approach the ego vehicle or conflict with the ego vehicle's driving path, and build different input sizes and permutation invariance into the decision sub-network. The outputs of encoder 1, encoder 2, and encoder 3 are all connected to the multi-head attention module to output the attention matrix. Thirdly, the output attention matrix is ​​connected to decoder 1. Encoder 1 consists of a fully connected layer.

[0063] Among them, the number of neurons in the fully connected layer is 64, and the activation function is the Sigmoid function.

[0064] Sub-step 3: Constructing a decision sub-network based on generative adversarial imitation learning

[0065] In an open, complex urban traffic environment with interference from multiple traffic targets, it is difficult to construct an accurate and comprehensive reward function, especially it is difficult to quantitatively describe the impact of uncertainty (such as sudden obstacles, traffic participants in visual blind spots, etc.) on driving safety. In order to reduce the impact of the uncertainty of the traffic environment and driving conditions on safe driving decisions, and to improve the effectiveness and reliability of driving decisions, the present invention uses a generative adversarial imitation learning subnetwork to learn driving strategies in driving behavior data sets and their generalized sample data, and then imitate safe driving behaviors under different driving conditions and driving conditions. The generative adversarial imitation learning subnetwork consists of two parts: a generator and a discriminator. The generator network and the discriminator network are constructed using deep neural networks respectively. The specific description is as follows:

[0066] (1) Build the generator

[0067] Construct a generator network. The input of the generator is the state space, and the output is the probability value of each action in the action space f = π(·|s; θ), where θ represents the parameters of the generator network. First, the state space is connected to the fully connected layers FC1 and FC2 in sequence to obtain feature F1. The state space is connected to the fully connected layers FC3 and FC4 in sequence to obtain feature F2. At the same time, the state space is connected to the convolutional layer C1 and the convolutional layer C2 in sequence to obtain feature F3. Then, features F1, F2, and F3 are connected to the merging layer, the fully connected layer FC5, and the Softmax activation function in sequence to obtain the output f = π(·|s; θ).

[0068] Among them, the number of neurons in the fully connected layers FC1, FC2, FC3, FC4 and FC5 is 64, the convolution kernel of the convolution layer C1 is 3×3, the stride is 2, and the convolution kernel of the convolution layer C2 is 3×3, the stride is 1.

[0069] (2) Constructing the discriminator

[0070] Construct a discriminator network. The input of the discriminator is the state space and the output is the vector The dimension is 6, where φ represents the parameters of the discriminator network. First, the state space is connected to the fully connected layers FC6 and FC7 in sequence to obtain feature F4. The state space is connected to the fully connected layers FC8 and FC9 in sequence to obtain feature F5. At the same time, the state space is connected to the convolutional layers C3 and C4 in sequence to obtain feature F6. Then, features F4, F5, and F6 are connected to the merging layer, the fully connected layer FC 10 , Sigmoid activation function, and get the output

[0071] Among them, the fully connected layers FC6, FC7, FC8, FC9 and FC 10 The number of neurons in the convolution layer is 64, the convolution kernel of the convolution layer C3 is 3×3, the stride is 2, and the convolution kernel of the convolution layer C4 is 3×3, the stride is 1.

[0072] Step 3: Training the safe driving decision model for operational vehicles

[0073] First, train the decision sub-network based on generative adversarial imitation learning. The goal of the generative adversarial imitation learning sub-network is to learn a generator network so that the discriminator cannot distinguish the driving actions generated by the generator from the actions in the driving behavior dataset. Specifically, it includes the following sub-steps:

[0074] Sub-step 1: In the driving behavior dataset In, initialize the generator network parameters θ0 and the discriminator network parameters ω0;

[0075] Sub-step 2: Perform L iterations to solve the problem, each iteration includes sub-steps 2.1 to 2.2, specifically:

[0076] Sub-step 2.1: Update the discriminator parameter ω using the gradient formula described by equation (13) i →ω i+1 :

[0077]

[0078] In the formula, Represents the gradient function of the neural network loss function with parameter ω;

[0079] Sub-step 2.2: Setting up the reward function Update the generator parameter θ using the trust region strategy optimization algorithm i →θ i+1 .

[0080] First, based on the above network training results, continue to train and build a decision subnetwork based on DDQN, which includes the following sub-steps:

[0081] Sub-step 3: Initialize the capacity of the experience reuse pool D to N;

[0082] Sub-step 4: Initialize the Q value corresponding to the action to a random value;

[0083] Sub-step 5: Perform M iterations to solve the problem, each iteration includes sub-steps 5.1 to 5.2, specifically:

[0084] Sub-step 5.1: Initialize state s0 and initialize policy parameter φ0;

[0085] Sub-step 5.2: Perform T iterations to solve the problem, each iteration includes sub-steps 5.21 to 5.27, specifically:

[0086] Sub-step 5.21: Randomly select a driving action;

[0087] Sub-step 5.22: Otherwise, select a t =max a Q * (φ(s t ),a;θ);

[0088] In the formula, Q * (·) represents the optimal action-value function, a t represents the action at time t;

[0089] Sub-step 5.23: Perform action a t , get the reward value r at time t t and the state s at time t+1 t+1 ;

[0090] Sub-step 5.24: Store the sample (φ) in the experience multiplexing pool D t ,a t ,r t ,φ t+1 );

[0091] Sub-step 5.25: Randomly draw a small batch of samples (φ j ,a j ,r j ,φ j+1 );

[0092] Sub-step 5.26: Calculate the iteration target using the following formula:

[0093]

[0094] In the formula, represents the weight of the target network at time t; γ represents the discount factor; argmax(·) represents the variable that maximizes the target function, y i represents the iteration target at time i, and p(s,a) represents the action distribution;

[0095] Sub-step 5.27: Use the following formula in (y i -Q(φ j ,a j ;θ)) 2 Perform gradient descent on:

[0096]

[0097] In the formula, Denote the parameter as θ i The gradient function of the neural network loss function, ε represents the probability of randomly selecting an action under the ε-greedy exploration strategy; θ i represents the parameter of iteration at time i, L i (θ i ) represents the loss function at time i, Q(s,a;θ i ) represents the action value function of the target network, and a′ represents all possible actions in state s′.

[0098] After the training of the safe driving decision model for operating vehicles is completed, the state space information collected by the sensor is input into the safe driving decision model, which can output high-level driving decisions such as turning, going straight, accelerating and decelerating in real time, and can effectively ensure the safe operation of operating vehicles in low-speed urban environments.

[0099] Beneficial effects: Compared with the general driving decision-making method, the method proposed by the present invention is more effective and reliable, which is specifically reflected in:

[0100] (1) The method proposed in the present invention can simulate the safe driving behavior of human drivers, provide a more reasonable and safe driving strategy for operating vehicles in low-speed urban environments, and achieve safe driving decisions for large operating vehicles with a highly human-like level, which can effectively ensure the driving safety of vehicles.

[0101] (2) The method proposed in the present invention comprehensively considers the impact of factors such as visual blind spots, sudden obstacles, and different driving conditions on driving safety, and conducts strategy learning and training in normal driving scenarios and edge scenarios, further improving the effectiveness and reliability of driving decisions.

[0102] (3) The method proposed in this paper introduces a multi-head attention mechanism, takes into account the dynamic interaction between the vehicle and surrounding traffic participants, and is able to handle safe driving decisions with variable input (dynamic changes in the number of surrounding traffic participants).

[0103] (4) The method proposed in the present invention does not need to consider complex vehicle dynamics equations and vehicle body parameters. The calculation method is simple and clear, and can output the safe driving decision strategy of large-scale operating vehicles in real time. The sensors used are low in cost, which is convenient for large-scale promotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 It is the technical roadmap of the present invention;

[0105] Figure 2 It is a schematic diagram of the strategy network structure based on the multi-head attention mechanism designed by the present invention;

[0106] Figure 3 It is a schematic diagram of the generator network structure designed by the present invention;

[0107] Figure 4 It is a schematic diagram of the discriminator network structure designed by the present invention. DETAILED DESCRIPTION

[0108] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0109] The present invention proposes a safe driving decision-making method for large commercial vehicles with a highly human-like level for an open urban traffic environment with multiple traffic target interference. First, the safe driving behavior of human drivers in an urban traffic environment is collected to construct a safe driving behavior data set. Secondly, a safe driving decision-making model for commercial vehicles based on multi-head attention is constructed. The model includes two sub-networks: a deep double Q network and a generative adversarial imitation learning sub-network. Among them, the deep double Q network learns safe driving strategies in edge scenarios such as dangerous scenarios and conflict scenarios through unsupervised learning; the generative adversarial imitation learning sub-network imitates safe driving behaviors under different driving conditions and driving conditions. Finally, the safe driving decision-making model is trained to obtain driving strategies under different driving conditions and driving conditions, and to achieve high-level decision output for the safe driving behavior of commercial vehicles. The method proposed in the present invention can simulate the safe driving behavior of human drivers, and takes into account the impact of factors such as visual blind spots and sudden obstacles on driving safety, providing more reasonable and safe driving strategies for large commercial vehicles, and realizing safe driving decisions for commercial vehicles in urban traffic environments. The technical route of the present invention is as follows: Figure 1 As shown, the specific steps are as follows:

[0110] Step 1: Collect safe driving behaviors of human drivers in urban traffic environments

[0111] In order to achieve driving decisions comparable to those of human drivers, the present invention collects safe driving behaviors under different driving conditions and driving conditions through actual road tests and driving simulations, and then constructs a data set that characterizes the safe driving behaviors of human drivers. Specifically, it includes the following five sub-steps:

[0112] Sub-step 1: Use millimeter-wave radar, 128-line lidar, visual sensor, Beidou sensor and inertial sensor to build a multi-dimensional target information synchronous acquisition system.

[0113] Sub-step 2: In a real urban environment, multiple drivers take turns driving an operating vehicle equipped with a multi-dimensional target information synchronous acquisition system.

[0114] Sub-step 3: Collect and process relevant data of various driving behaviors such as lane changing, lane keeping, vehicle following, acceleration and deceleration, and obtain multi-source heterogeneous description data of each driving behavior, such as obstacle distances in multiple directions measured by radar or visual sensors, position, speed, acceleration and yaw angular velocity measured by Beidou sensors and inertial sensors, and steering wheel angle measured by vehicle-mounted sensors.

[0115] Sub-step 4: In order to simulate safe driving behaviors in edge scenarios such as dangerous scenarios and conflict scenarios, a virtual city scenario based on hardware-in-the-loop simulation is built. The constructed urban traffic scenarios include the following three categories:

[0116] (1) When the vehicle is driving, there is a traffic participant approaching laterally in front of the vehicle (i.e., encountering an obstacle suddenly);

[0117] (2) When the vehicle is turning, there are stationary traffic participants in the vehicle's blind spot;

[0118] (3) When the vehicle is turning, there are moving traffic participants in the vehicle’s blind spot.

[0119] In the above traffic scenarios, there are various road network structures (straight roads, curves and intersections) and multiple types of traffic participants (commercial vehicles, passenger cars, non-motor vehicles and pedestrians).

[0120] Multiple drivers drive the operating vehicles in the virtual scene using real controllers (steering wheel, accelerator and brake pedal) to collect information such as the vehicle's lateral and longitudinal position, lateral and longitudinal speed, lateral and longitudinal acceleration, relative distance and relative speed to surrounding traffic participants.

[0121] Sub-step 5: Based on the data collected from the real urban environment and the driving simulation environment, a driving behavior dataset for safe driving decision learning is constructed, which can be specifically expressed as:

[0122]

[0123] In the formula, X represents a binary group covering states and actions, that is, a dataset constructed to characterize the safe driving behavior of human drivers, (s j ,a j ) represents the state-action pair at time j, where s j represents the state at time j, a j represents the action at time j, i.e. the human driver’s action based on state s j The action taken, n is the number of "state-action" pairs in the database.

[0124] Step 2: Construct a safe driving decision model for commercial vehicles based on multi-head attention

[0125] In order to achieve safe driving decisions for large commercial vehicles in low-speed urban environments, the present invention comprehensively considers the impact of factors such as visual blind spots, sudden obstacles, and driving conditions on driving safety, and establishes a safe driving decision-making model for commercial vehicles. Considering that deep reinforcement learning combines the perception ability of deep learning with the decision-making ability of reinforcement learning, and explores the traffic environment through unsupervised learning, the present invention uses deep reinforcement learning to learn safe driving strategies in edge scenarios such as dangerous scenarios and conflict scenarios. In addition, considering that imitation learning has the ability to emulate role models, the present invention uses imitation learning to simulate the safe driving behavior of human drivers under different driving conditions and driving conditions. Therefore, the constructed safe driving decision model consists of two parts, which are specifically described as follows:

[0126] Sub-step 1: Define the basic parameters of the safe driving decision model

[0127] Firstly, the safe driving decision-making problem in a low-speed urban environment is transformed into a finite Markov decision process. Secondly, the basic parameters of the safe driving decision-making model are defined.

[0128] (1) Define the state space

[0129] In order to describe the motion state of the vehicle and nearby traffic participants, the present invention uses time series data and occupancy grid graph to construct the state space. The specific description is as follows:

[0130] S t =[S1(t),S2(t),S3(t)] (2)

[0131] In the formula, S t represents the state space at time t, S1(t) and S2(t) represent the state space associated with the time series data at time t, and S3(t) represents the state space associated with the occupancy grid at time t.

[0132] First, the motion state of the vehicle is described using continuous position, velocity, acceleration, and heading information:

[0133] S1(t)=[p x ,p y ,v x ,v y ,a x ,a y ,θ s ] (3)

[0134] In the formula, p x ,p y Respectively represent the lateral position and longitudinal position of the vehicle, in meters, v x ,v y They represent the lateral speed and longitudinal speed of the vehicle, in meters per second, and ax ,a y They represent the lateral acceleration and longitudinal acceleration of the vehicle, respectively, in meters per second squared, and θ s Indicates the heading angle of the vehicle in degrees.

[0135] Secondly, the relative motion state information between the vehicle and surrounding traffic participants is used to describe the motion state of surrounding traffic participants:

[0136]

[0137] In the formula, They represent the relative distance, relative speed and acceleration between the ego-vehicle and the i-th traffic participant, in meters, meters per second and meters per second squared, respectively.

[0138] In the existing state space definition methods, fixed coding methods are often used, that is, the number of surrounding traffic participants considered is fixed. However, in actual urban traffic scenarios, the number and positions of traffic participants around operating vehicles are constantly changing, and side collisions caused by sudden obstacles and visual blind spots need to be specially considered. Although the fixed coding method can achieve effective state representation, the number of traffic participants considered is limited (the minimum amount of information required to represent the scene is used), and it is impossible to accurately and comprehensively describe the impact of all surrounding traffic participants on the driving safety of operating vehicles.

[0139] Finally, in order to more vividly describe the relative position relationship between the vehicle and surrounding traffic participants and improve the reliability and effectiveness of decision-making, the present invention grids the road area into several a×b grid areas, and abstracts the road area and vehicle targets into a grid map, that is, the "existence" grid map S3(t) used to describe the relative position relationship. Where a represents the length of the grid area, and b represents the width of the grid area.

[0140] The "existence" grid map contains four attributes, including grid coordinates, whether there is a vehicle, the type of the corresponding vehicle, and the distance from the left and right lane lines. Among them, the grid without traffic participants is set to "0", and the grid with traffic participants is set to "1". The position distribution of this grid and the grid where the vehicle is located is used to describe the relative distance between the two vehicles.

[0141] (2) Define the action space

[0142] Define the action space using lateral and longitudinal driving actions:

[0143] A t =[a left ,a straight ,a right ,a accel ,a cons ,a decel] (5)

[0144] In the formula, A t represents the action space at time t, a left ,a straight ,a right They represent left turn, straight ahead and right turn respectively. accel ,a cons ,a decel They represent acceleration, constant speed, and deceleration respectively.

[0145] (3) Define the reward function

[0146] R t =r1+r2+r3 (6)

[0147] In the formula, R t represents the reward function at time t, r1, r2, r3 represent the forward collision avoidance reward function, the backward collision avoidance reward function, and the side collision avoidance reward function, respectively, which can be obtained by equations (7), (8), and (9).

[0148]

[0149]

[0150]

[0151] Where, TTC represents the time when the ego vehicle collides with the obstacle in front, which can be obtained by dividing the distance between the ego vehicle and the obstacle in front by the relative speed. thr Indicates the distance to collision time threshold, RTTC indicates the rearward collision time, RTTC thr Represents the rear collision time threshold, in seconds, x lat represents the distance between the vehicle and the traffic participants on both sides, x min represents the minimum lateral safety distance in meters. β1, β2, and β3 represent the weight coefficients of the forward collision avoidance reward function, the backward collision avoidance reward function, and the lateral collision avoidance reward function, respectively.

[0152] Sub-step 2: Construct a DDQN-based decision sub-network

[0153] Considering that the Double Deep Q Network (DDQN) improves the efficiency of data utilization by using the experience reuse pool, and can avoid parameter oscillation or divergence, it can reduce the negative learning effect caused by overestimation in the Q learning network. Therefore, the present invention uses the deep double Q network to learn safe driving strategies in edge scenarios.

[0154] Unlike processing a state space with fixed dimensions, processing feature information covering all surrounding traffic participants requires stronger feature extraction capabilities. Considering that the attention mechanism can capture richer feature information (the dependency between the vehicle and surrounding traffic participants), the present invention designs a policy network based on a multi-head attention mechanism. In addition, considering that driving decisions are only related to the motion state of the vehicle and surrounding traffic participants, and should not be affected by the order of traffic participants in the state space, the present invention uses a position encoding method (literature: Vaswani, Ashish, et al. "Attention is all you need." Advances in Neural Information Processing Systems. 2017.) to build permutation invariance into the decision subnetwork.

[0155] The attention layer can be expressed as:

[0156] MultiHead(Q,K,V)=Concat(head1,...,head h )W O (10)

[0157] In the formula, MultiHead(Q,K,V) represents the multi-head attention value, Q represents the query vector, K represents the key vector, and the dimension is d k , V represents the value vector, dimension is d v , W O Represents the parameter matrix that needs to be learned, head h represents the hth head in the multi-head attention. In the present invention, h=2, which can be calculated by the following formula:

[0158]

[0159]

[0160] In the formula, Attention(Q,K,V) represents the output attention matrix, Represents the parameter matrix that needs to be learned.

[0161] Construct a decision sub-network based on a deep double Q network, such as Figure 2 The specific description is as follows.

[0162] First, the state space S t They are connected to encoder 1, encoder 2 and encoder 3 respectively. Encoder 1 consists of two fully connected layers and outputs the vehicle motion state encoding. The structure of encoder 2 is the same as encoder 1 and outputs the relative motion state encoding. Encoder 3 consists of two convolutional layers and outputs the occupation grid map encoding.

[0163] The number of neurons in the fully connected layer is 64, the activation function is the Tanh function, the convolution kernel of the convolution layer is 3×3, and the step size is 2.

[0164] Secondly, the multi-head attention mechanism is used to analyze the dependency between the ego vehicle and surrounding traffic participants, so that the decision sub-network can notice traffic participants that suddenly approach the ego vehicle or conflict with the ego vehicle's driving path, and build different input sizes and permutation invariance into the decision sub-network. The outputs of encoder 1, encoder 2, and encoder 3 are all connected to the multi-head attention module to output the attention matrix. Thirdly, the output attention matrix is ​​connected to decoder 1. Encoder 1 consists of a fully connected layer.

[0165] Among them, the number of neurons in the fully connected layer is 64, and the activation function is the Sigmoid function.

[0166] Sub-step 3: Constructing a decision sub-network based on generative adversarial imitation learning

[0167] In an open, complex urban traffic environment with interference from multiple traffic targets, it is difficult to construct an accurate and comprehensive reward function, especially it is difficult to quantitatively describe the impact of various uncertainties (such as sudden obstacles, traffic participants in visual blind spots, etc.) on driving safety. In order to reduce the impact of the uncertainty of the traffic environment and driving conditions on safe driving decisions, and to improve the effectiveness and reliability of driving decisions, the present invention uses a generative adversarial imitation learning subnetwork to learn driving strategies in driving behavior data sets and their generalized sample data, and then imitate safe driving behaviors under different driving conditions and driving conditions. The generative adversarial imitation learning subnetwork consists of two parts: a generator and a discriminator. The generator network and the discriminator network are constructed using deep neural networks respectively. The specific description is as follows:

[0168] (1) Build the generator

[0169] Build as Figure 3 The generator network shown. The input of the generator is the state space, and the output is the probability value of each action in the action space f = π(·|s; θ), where θ represents the parameters of the generator network. First, the state space is connected to the fully connected layers FC1 and FC2 in sequence to obtain feature F1. The state space is connected to the fully connected layers FC3 and FC4 in sequence to obtain feature F2. At the same time, the state space is connected to the convolutional layer C1 and the convolutional layer C2 in sequence to obtain feature F3. Then, the features F1, F2 and F3 are connected to the merging layer, the fully connected layer FC5, and the Softmax activation function in sequence to obtain the output f = π(·|s; θ).

[0170] Among them, the number of neurons in the fully connected layers FC1, FC2, FC3, FC4 and FC5 is 64, the convolution kernel of the convolution layer C1 is 3×3, the stride is 2, and the convolution kernel of the convolution layer C2 is 3×3, the stride is 1.

[0171] (2) Constructing the discriminator

[0172] Build as Figure 4 The discriminator network shown. The input of the discriminator is the state space and the output is the vector The dimension is 6, where φ represents the parameters of the discriminator network. First, the state space is connected to the fully connected layers FC6 and FC7 in sequence to obtain feature F4. The state space is connected to the fully connected layers FC8 and FC9 in sequence to obtain feature F5. At the same time, the state space is connected to the convolutional layers C3 and C4 in sequence to obtain feature F6. Then, features F4, F5, and F6 are connected to the merging layer, the fully connected layer FC 10 , Sigmoid activation function, and get the output

[0173] Among them, the fully connected layers FC6, FC7, FC8, FC9 and FC 10 The number of neurons in the convolution layer is 64, the convolution kernel of the convolution layer C3 is 3×3, the stride is 2, and the convolution kernel of the convolution layer C4 is 3×3, the stride is 1.

[0174] Step 3: Training the safe driving decision model for operational vehicles

[0175] First, train the decision sub-network based on generative adversarial imitation learning. The goal of the generative adversarial imitation learning sub-network is to learn a generator network so that the discriminator cannot distinguish the driving actions generated by the generator from the actions in the driving behavior dataset. Specifically, it includes the following sub-steps:

[0176] Sub-step 1: In the driving behavior dataset In, initialize the generator network parameters θ0 and the discriminator network parameters ω0;

[0177] Sub-step 2: Perform L iterations to solve the problem, each iteration includes sub-steps 2.1 to 2.2, specifically:

[0178] Sub-step 2.1: Update the discriminator parameter ω using the gradient formula described by equation (13) i →ω i+1 :

[0179]

[0180] In the formula, Represents the gradient function of the neural network loss function with parameter ω;

[0181] Sub-step 2.2: Setting up the reward function Update the generator parameter θ using the trust region strategy optimization algorithm i →θ i+1 .

[0182] First, based on the above network training results, continue to train and build a decision subnetwork based on DDQN, which includes the following sub-steps:

[0183] Sub-step 3: Initialize the capacity of the experience reuse pool D to N;

[0184] Sub-step 4: Initialize the Q value corresponding to the action to a random value;

[0185] Sub-step 5: Perform M iterations to solve the problem, each iteration includes sub-steps 5.1 to 5.2, specifically:

[0186] Sub-step 5.1: Initialize state s0 and initialize policy parameter φ0;

[0187] Sub-step 5.2: Perform T iterations to solve the problem, each iteration includes sub-steps 5.21 to 5.27, specifically:

[0188] Sub-step 5.21: Randomly select a driving action;

[0189] Sub-step 5.22: Otherwise, select a t =max a Q * (φ(s t ),a;θ);

[0190] In the formula, Q * (·) represents the optimal action-value function, a t represents the action at time t;

[0191] Sub-step 5.23: Perform action a t , get the reward value r at time t t and the state s at time t+1 t+1 ;

[0192] Sub-step 5.24: Store the sample (φ) in the experience multiplexing pool D t ,a t ,r t ,φ t+1 );

[0193] Sub-step 5.25: Randomly draw a small batch of samples (φ j ,a j ,r j ,φ j+1 );

[0194] Sub-step 5.26: Calculate the iteration target using the following formula:

[0195]

[0196] In the formula, represents the weight of the target network at time t; γ represents the discount factor; argmax(·) represents the variable that maximizes the target function, y i represents the iteration target at time i, and p(s,a) represents the action distribution;

[0197] Sub-step 5.27: Use the following formula in (y i -Q(φ j ,a j ;θ)) 2 Perform gradient descent on:

[0198]

[0199] In the formula, Denote the parameter as θ i The gradient function of the neural network loss function, ε represents the probability of randomly selecting an action under the ε-greedy exploration strategy; θ i represents the parameter of iteration at time i, L i (θ i ) represents the loss function at time i, Q(s,a;θ i ) represents the action value function of the target network, and a′ represents all possible actions in state s′.

[0200] After the training of the safe driving decision model for operating vehicles is completed, the state space information collected by the sensor is input into the safe driving decision model, which can output high-level driving decisions such as turning, going straight, accelerating and decelerating in real time, and can effectively ensure the safe operation of operating vehicles in low-speed urban environments.

Claims

1. A safe driving decision-making method for large commercial vehicles in low-speed urban environments. First, collect the safe driving behaviors of human drivers in urban traffic environments to construct a safe driving behavior dataset; second, build a safe driving decision-making model for commercial vehicles based on multi-head attention; the model includes two sub-networks: a deep dual-Q network and a generative adversarial imitation learning network; among them, The deep dual-Q network learns safe driving strategies in dangerous and conflict scenarios through unsupervised learning. The generative adversarial imitation learning sub-network imitates the safe driving behavior of human drivers under different driving conditions and driving conditions. Finally, the safe driving decision model is trained to obtain driving strategies under different driving conditions and driving conditions, and to achieve high-level decision output on the safe driving behavior of operating vehicles; the characteristics are: Step 1: Collect safe driving behaviors of human drivers in urban traffic environments Through actual road tests and driving simulations, we collect safe driving behaviors under different driving conditions and driving scenarios, and then build a data set that characterizes the safe driving behaviors of human drivers. This includes the following four sub-steps: Sub-step 1: Use millimeter wave radar, 128-line laser radar, visual sensor, Beidou sensor and inertial sensor to build a multi-dimensional target information synchronous acquisition system; Sub-step 2: In a real urban environment, multiple drivers sequentially drive an operating vehicle equipped with a multi-dimensional target information synchronous acquisition system to collect and process relevant data on the drivers' lane changing, lane keeping, vehicle following, acceleration and deceleration, and obtain multi-source heterogeneous description data of each driving behavior, including obstacle distances in multiple directions measured by radar or visual sensors, position, speed, acceleration and yaw angular velocity measured by Beidou sensors and inertial sensors, and steering wheel angle measured by vehicle-mounted sensors; Sub-step 3: In order to simulate safe driving behaviors in dangerous and conflict scenarios, a virtual city scene based on hardware-in-the-loop simulation is constructed; the constructed urban traffic scenes include the following three categories: (1) When the vehicle is driving, there is a traffic participant approaching laterally in front of the vehicle, i.e., a sudden obstacle; (2) When the vehicle is turning, there are stationary traffic participants in the vehicle's blind spot; (3) When the vehicle is turning, there are moving traffic participants in the vehicle's blind spot; In the above traffic scenarios, there are various road network structures, including straight roads, curves and intersections, and multiple types of traffic participants, including commercial vehicles, passenger cars, non-motor vehicles and pedestrians; Multiple drivers use real controllers with steering wheels, accelerators and brake pedals to drive commercial vehicles in virtual scenes, and collect information on the lateral and longitudinal positions, lateral and longitudinal speeds, lateral and longitudinal accelerations, and the relative distance and relative speed to surrounding traffic participants; Sub-step 4: Based on the data collected from the real urban environment and the driving simulation environment, a driving behavior dataset for safe driving decision learning is constructed, which is specifically expressed as: In the formula, X represents a binary group covering states and actions, that is, a dataset constructed to characterize the safe driving behavior of human drivers, (s j ,a j ) represents the "state-action" pair at time j, where s j represents the state at time j, a j represents the action at time j, i.e. the human driver’s action based on state s j The action taken, n represents the number of "state-action" pairs in the database; Step 2: Construct a safe driving decision model for commercial vehicles based on multi-head attention Deep reinforcement learning is used to learn safe driving strategies in dangerous and conflict scenarios. In addition, considering that imitation learning has the ability to imitate role models, imitation learning is used to simulate the safe driving behavior of human drivers under different driving conditions and driving conditions. Therefore, the constructed safe driving decision model consists of two parts, which are described as follows: Sub-step 1: Define the basic parameters of the safe driving decision model Firstly, the safe driving decision-making problem in a low-speed urban environment is transformed into a finite Markov decision process; secondly, the basic parameters of the safe driving decision-making model are defined; (1) Define the state space In order to describe the motion state of the ego vehicle and nearby traffic participants, the state space is constructed using time series data and occupancy grid graphs; the specific description is as follows: S t =[S1(t),S2(t),S3(t)] (2) In the formula, S t represents the state space at time t, S1(t) represents the state space of the ego vehicle related to the time series data at time t, S2(t) represents the state space of the surrounding traffic participants related to the time series data at time t, and S3(t) represents the state space related to the occupancy grid at time t; First, the motion state of the vehicle is described using continuous position, velocity, acceleration, and heading information: S1(t)=[p x ,p y ,v x ,v y ,a x ,a y ,θ s ] (3) In the formula, p x ,p y Respectively represent the lateral position and longitudinal position of the vehicle, in meters, v x ,v y They represent the lateral speed and longitudinal speed of the vehicle, in meters per second, and a x ,a y They represent the lateral acceleration and longitudinal acceleration of the vehicle, respectively, in meters per second squared, and θ s Indicates the heading angle of the vehicle, in degrees; Secondly, the relative motion state information between the vehicle and surrounding traffic participants is used to describe the motion state of surrounding traffic participants: In the formula, a i They represent the relative distance, relative speed and acceleration between the ego vehicle and the i-th traffic participant, respectively, in units of meters, meters per second and meters per second squared; Finally, the road area is rasterized and divided into several p×q grid areas, and the road area and vehicle targets are abstracted into a grid map, that is, the "existence" grid map S3(t) used to describe the relative position relationship; where p represents the length of the grid area and q represents the width of the grid area; The "existence" grid map contains four attributes, including grid coordinates, whether there is a vehicle, the type of the corresponding vehicle, and the distance to the left and right lane lines. Among them, the grid without traffic participants is set to "0", and the grid with traffic participants is set to "1". The position distribution of this grid and the grid where the vehicle is located is used to describe the relative distance between the two vehicles. (2) Define the action space Define the action space using lateral and longitudinal driving actions: A t =[a left ,a straight ,a right ,a accel ,a cons ,a decel ] (5) In the formula, A t represents the action space at time t, a left ,a straight ,a right They represent left turn, straight ahead and right turn respectively. accel ,a cons ,a decel They represent acceleration, uniform speed and deceleration respectively; (3) Define the reward function R t ＝r1+r2+r3 (6) In the formula, R t represents the reward function at time t, r1, r2, r3 represent the forward collision avoidance reward function, the backward collision avoidance reward function, and the side collision avoidance reward function, respectively, which can be obtained by equations (7), (8), and (9); Where, TTC represents the time when the ego vehicle collides with the obstacle in front, which can be obtained by dividing the distance between the ego vehicle and the obstacle in front by the relative speed. thr Indicates the distance to collision time threshold, RTTC indicates the rearward collision time, RTTC thr Represents the rear collision time threshold, in seconds, x lat represents the distance between the vehicle and the traffic participants on both sides, x min represents the minimum lateral safety distance, all in meters, β1, β2, β3 represent the weight coefficients of the forward collision avoidance reward function, the backward collision avoidance reward function, and the lateral collision avoidance reward function respectively; Sub-step 2: Construct a decision sub-network based on a deep double Q network Using deep dual-Q networks to learn safe driving strategies in edge scenarios; Design a policy network based on a multi-head attention mechanism; in addition, use the position encoding method to build permutation invariance into the decision subnetwork; The attention layer can be expressed as: MultiHead(Q,K,V)=Concat(head1,...,head h )W O (10) In the formula, MultiHead(Q,K,V) represents the multi-head attention value, Q represents the query vector, K represents the key vector, and the dimension is d k , V represents the value vector, W O Represents the parameter matrix that needs to be learned, head h represents the h-th head in the multi-head attention, h = 2, and is calculated by the following formula: In the formula, Attention(Q,K,V) represents the output attention matrix, Represents the parameter matrix that needs to be learned; Construct a decision subnetwork, as described below; First, the state space S t They are connected to encoder 1, encoder 2 and encoder 3 respectively; encoder 1 consists of two fully connected layers and outputs the vehicle motion state code; the structure of encoder 2 is the same as encoder 1, and outputs the relative motion state code; encoder 3 consists of two convolutional layers and outputs the occupancy grid map code; Among them, the number of neurons in the fully connected layer is 64, and the activation function is the Tanh function; the convolution kernel of the convolution layer is 3×3, and the step size is 2; Secondly, the multi-head attention mechanism is used to analyze the dependency between the ego vehicle and surrounding traffic participants, so that the decision sub-network can notice traffic participants that suddenly approach the ego vehicle or conflict with the ego vehicle's driving path, and build different input sizes and permutation invariance into the decision sub-network; the outputs of encoder 1, encoder 2, and encoder 3 are all connected to the multi-head attention module to output the attention matrix; thirdly, the output attention matrix is ​​connected to decoder 1; encoder 1 consists of a fully connected layer; Among them, the number of neurons in the fully connected layer is 64, and the activation function is the Sigmoid function; Sub-step 3: Constructing a decision sub-network based on generative adversarial imitation learning The generative adversarial imitation learning sub-network is used to learn the driving strategies in the driving behavior dataset and its generalized sample data, thereby imitating the safe driving behavior of human drivers under different driving conditions and driving conditions; the generative adversarial imitation learning sub-network consists of two parts: the generator and the discriminator, which are constructed using deep neural networks respectively; the specific description is as follows: (1) Build the generator Construct a generator network; the input of the generator is the state space, and the output is the probability value of each action in the action space f = π (·s; θ), where θ represents the parameters of the generator network. First, the state space is connected to the fully connected layers FC1 and FC2 in sequence to obtain feature F1; the state space is connected to the fully connected layers FC3 and FC4 in sequence to obtain feature F2; at the same time, the state space is connected to the convolutional layers C1 and C2 in sequence to obtain feature F3; then, the features F1, F2 and F3 are connected to the merging layer, the fully connected layer FC5, and the softmax() activation function in sequence to obtain the output f = π (·s; θ); Among them, the number of neurons in the fully connected layers FC1, FC2, FC3, FC4 and FC5 is 64, the convolution kernel of the convolution layer C1 is 3×3, the stride is 2, and the convolution kernel of the convolution layer C2 is 3×3, the stride is 1; (2) Constructing the discriminator Construct a discriminator network; the input of the discriminator is the state space and the output is a vector The dimension is 6, where φ represents the parameters of the discriminator network. First, the state space is connected to the fully connected layers FC6 and FC7 in sequence to obtain feature F4. The state space is connected to the fully connected layers FC8 and FC9 in sequence to obtain feature F5. At the same time, the state space is connected to the convolutional layers C3 and C4 in sequence to obtain feature F6. Then, features F4, F5, and F6 are connected to the merging layer, the fully connected layer FC 10 , Sigmoid activation function, and get the output Among them, the fully connected layers FC6, FC7, FC8, FC9 and FC 10 The number of neurons in the convolution layer is 64, the convolution kernel of the convolution layer C3 is 3×3, the stride is 2, and the convolution kernel of the convolution layer C4 is 3×3, the stride is 1; Step 3: Training the safe driving decision model for operational vehicles First, train the decision subnetwork based on generative adversarial imitation learning. The goal of the generative adversarial imitation learning subnetwork is to learn a generator network so that the discriminator cannot distinguish the driving actions generated by the generator from the actions in the driving behavior dataset. Specifically, it includes the following sub-steps: Sub-step 1: In the driving behavior dataset In, initialize the generator network parameters θ0 and the discriminator network parameters ω0; Sub-step 2: Perform L iterations to solve the problem, each iteration includes sub-steps 2.1 to 2.2, specifically: Sub-step 2.1: Update the discriminator parameter ω using the gradient formula described by equation (13) i →ω i+1 : In the formula, represents the gradient function of the neural network loss function with parameter ω, Sub-step 2.2: Setting up the reward function Update the generator parameter θ using the trust region strategy optimization algorithm i →θ i+1 ; First, based on the network training results, continue to train and build a decision subnetwork based on DDQN, which includes the following sub-steps: Sub-step 3: Initialize the capacity of the experience reuse pool D to N; Sub-step 4: Initialize the Q value corresponding to the action to a random value; Sub-step 5: Perform M iterations to solve the problem, each iteration includes sub-steps 5.1 to 5.2, specifically: Sub-step 5.1: Initialize state s0 and initialize strategy parameter φ0; Sub-step 5.2: Perform T iterations to solve the problem, each iteration includes sub-steps 5.21 to 5.27, specifically: Sub-step 5.21: Randomly select a driving action; Sub-step 5.22: Otherwise, select a t =max a [Q * (φ(s t ),a;θ)]; In the formula, Q * (·) represents the optimal action-value function, a t represents the action at time t, Sub-step 5.23: Perform action a t , get the reward value r at time t t and the state s at time t+1 t+1 ; Sub-step 5.24: Store the sample (φ) in the experience multiplexing pool D t ,a t ,r t ,φ t+1 ), Sub-step 5.25: Randomly draw a small batch of samples (φ j ,a j ,r j ,φ j+1 ), Sub-step 5.26: Calculate the iteration target using the following formula: In the formula, represents the weight of the target network at time t; γ represents the discount factor; argmax a (·) represents the variable that maximizes the objective function, y i represents the iteration target at time i, Sub-step 5.27: Use the following formula in (y i -Q(φ j ,a j ;θ)) 2 Perform gradient descent on: In the formula, Denote the parameter as θ i The gradient function of the neural network loss function, ε represents the probability of randomly selecting an action under the ε-greedy exploration strategy; θ i represents the parameter of iteration at time i, L i (θ i ) represents the loss function at time i, a' represents all possible actions in state s', After the training of the safe driving decision model for operating vehicles is completed, the state space information collected by the sensor is input into the safe driving decision model, which can output high-level driving decisions such as turning, going straight, and accelerating and decelerating in real time, effectively ensuring the safe operation of operating vehicles in low-speed urban environments.

Images