Intelligent vehicle decision-making method for no-signal crossroad

Abstract

The invention belongs to the technical field of intelligent driving decision, and discloses a no-signal crossroad intelligent vehicle decision method, which comprises the following steps: constructing a no-signal crossroad intelligent vehicle decision model including a strategy network and a value network; each of the strategy network and the value network is composed of an LSTM neural network and a full connection layer; the method comprises the following steps of: forming state parameters by self-vehicle information, traffic participation vehicle information, relative position information of a traffic intersection and a self-vehicle and relative position information of a target task point and the self-vehicle from a # imgabs0 # moment to a # imgabs1 # moment as input variables of a strategy network; using an accelerator control signal, a steering wheel angle control signal and a brake control signal corresponding to the # imgabs2 moment state parameter to form an action parameter as an output variable of the strategy network; optimizing the decision model by adopting a PPO algorithm and taking the minimum loss function of the strategy network and the value network as an optimization target to obtain an optimal decision model; and outputting a vehicle decision by using the optimal decision model.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent driving decision-making, and in particular relates to an intelligent vehicle decision-making method for an unsignaled intersection. Background Art

[0002] In urban roads, unsignalized intersections are common traffic scenes. Such intersections do not have traffic lights or other control devices, and rely on mutual cooperation and real-time judgment between drivers to complete vehicle traffic. However, there are significant challenges in vehicle traffic at unsignalized intersections, and traditional rule-based vehicle decision-making methods are difficult to effectively cope with highly complex traffic environments. Especially in the case of large traffic volume, unclear priority, or significant differences in driving behavior, it is easy to lead to low traffic efficiency, limited traffic capacity, and even cause safety accidents. In addition, due to the dynamic characteristics of intersections, the judgment and decision-making process of drivers at intersections is full of uncertainty, which further exacerbates the complexity of traffic management. In order to improve the traffic efficiency and safety of unsignalized intersections, automated decision-making based on intelligent driving has become a research hotspot. In traditional technologies, rule algorithms or trajectory planning are often used for intersection decision-making, mainly including rule-based decision-making methods, trajectory optimization methods, and machine learning-based prediction models. Although these methods have certain effects in specific scenarios, they often show problems of being too conservative or too radical in unsignalized intersections, and it is difficult to strike a balance between efficiency and safety. In addition, traditional methods usually rely on manually configured rules or limited data samples, which are difficult to adapt to the dynamic and complex traffic characteristics of intersections.

[0003] In recent years, with the development of deep learning and reinforcement learning, more and more research has begun to explore the use of data-driven methods. By simulating a large number of real-world scenarios, reinforcement learning algorithms can gradually learn the optimal decision-making strategy. However, when dealing with intersection decision-making problems, existing reinforcement learning algorithms usually need to calculate conflict points. Since the accuracy of conflict point calculation is usually based on the accuracy of the model's prediction of the participant's vehicle trajectory, the existing reinforcement learning algorithms will lead to problems such as the vehicle trajectory prediction process being too simple, resulting in difficulty in algorithm convergence, relatively high computational overhead costs, and overly conservative vehicle decision strategies that reduce traffic efficiency. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the prior art and provide an intelligent vehicle decision-making method for an unsignalized intersection, which can reduce the conservatism of the decision-making model in the decision-making process and improve the vehicle traffic efficiency at the unsignalized intersection while ensuring safety.

[0005] The technical solution provided by the present invention is: An intelligent vehicle decision-making method for an unsignaled intersection, comprising: Constructing an intelligent vehicle decision model for an unsignalized intersection, the decision model comprising: a strategy network and a value network; Wherein, the strategy network and the value network are both composed of LSTM neural networks and fully connected layers; The information of the ego vehicle from time tn to time t, the information of the participating vehicles in the traffic, the relative position information of the traffic intersection and the ego vehicle, and the relative position information of the target task point and the ego vehicle form the state parameters as the input variables of the strategy network, and the throttle control signal, steering wheel angle control signal and brake control signal corresponding to the state parameters at time t form the action parameters as the output variables of the strategy network; The state parameter and the action parameter are input into the value network, and the value network outputs the value corresponding to the state parameter and the action parameter; The PPO algorithm is used to construct the loss functions of the policy network and the value network respectively, and the decision model is trained. The parameters of the policy network and the value network are optimized respectively with the minimum loss function as the optimization goal to obtain the optimal decision model; When the vehicle travels to an unsignaled intersection, state parameters are acquired in real time and input into the optimal decision model, and the optimal decision model outputs an optimal throttle control signal, a steering wheel angle control signal, and a brake control signal.

[0006] Preferably, the loss function of the strategy network is: ; in, is the clipping function, is the importance sampling ratio, are policy network parameters, is the strategy advantage function, and They are The state at the moment and the corresponding action, represents the clipping threshold; Expressing expectation.

[0007] Preferably, the loss function of the value network is: ; in, is the state value function, for the input state , the output is the value of the current state; for The actual value of the moment.

[0008] Preferably, the calculation formula of the actual value is: ; in, is the value advantage function; ; , , They are time, time, time, TD error at the moment; The calculation formula of TD error at time is: ; In the formula, for The actual reward of the moment, The corresponding state of the value network output The value of is the attenuation factor.

[0009] Preferably, the actual reward is calculated by the following reward function: ; in, The reward value for completing the task. is the collision penalty value, is the progressive distance reward value, is the progressive penalty value of collision risk, is the reasonable speed reward value, is the stability reward value.

[0010] Preferably, the calculation formula of the progressive reward is: ; in, is the value of the current time step, is the normalized distance, , , is the weight coefficient; is the distance from the vehicle to the target position at the initial moment, for The distance from the vehicle to the target location at any moment.

[0011] Preferably, the method for calculating the progressive penalty value of collision risk comprises the following steps: Step 1: According to the vehicle position ( , ) and the positions of participating vehicles in the traffic ( , ) Calculate the relative position vector ; According to the vehicle speed ( , ) and the speed of participating vehicles in the traffic ( , ) Calculate the relative velocity vector ; Step 2: Calculate the relative speed of the vehicle along the relative position direction :

[0012] Step 3: Calculate the estimated collision time :

[0013] Step 4: Determine the progressive penalty value of collision risk based on the expected collision time: ; in, , , , is the adjustment parameter and satisfies .

[0014] Preferably, the number of layers of the LSTM neural network used in the strategy network and the value network is 1, and the number of hidden units is 128.

[0015] Preferably, the number of neurons in the fully connected layer used by the strategy network and the value network is 64.

[0016] Preferably, the activation function used by the fully connected layer of the strategy network is Tanh, and the activation function used by the fully connected layer of the value network is ReLU.

[0017] The beneficial effects of the present invention are: The intelligent vehicle decision-making method for an unsignaled intersection provided by the present invention can reduce the conservatism of the decision-making model in the decision-making process, and improve the vehicle traffic efficiency at an unsignaled intersection while ensuring safety. At the same time, by designing a reward function that is more suitable for the intersection scenario, the present invention can improve the decision-making system's ability to understand the task, increase the model convergence speed, save storage and computing resources, reduce the model computing overhead, and further improve the exploratory nature and risk tolerance of the decision-making system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The figure is a flow chart of the intelligent vehicle decision-making method at an unsignalized intersection according to the present invention.

[0019] Figure 2 It is a schematic diagram of the intersection decision model based on Markov chain described in the present invention.

[0020] Figure 3 It is a schematic diagram of the relative speed calculation process of a vehicle along the relative position direction according to the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0022] like Figure 1 As shown, the present invention provides an intelligent vehicle decision-making method at an unsignalized intersection, including the following implementation process.

[0023] 1. Information Acquisition Traffic information at unsignaled intersections is obtained through the collaboration of on-board sensors and communication equipment, including: the vehicle obtains its own global status information such as position, speed and heading angle through GPS and inertial measurement unit (IMU); uses laser radar (LiDAR) and cameras to build a three-dimensional perception map of the surrounding environment to identify intersection coordinates and surrounding vehicles; uses millimeter-wave radar to accurately measure the relative distance and relative speed of other vehicles; and combines V2X (Vehicle-to-Everything) communication technology to obtain status information (such as position, speed, intention) of other vehicles to enhance perception.

[0024] 2. Constructing a vehicle decision-making model for unsignalized intersections like Figure 2 As shown, in this embodiment, a vehicle decision model for an unsignalized intersection is constructed based on a Markov chain, and the specific process is as follows.

[0025] 2.1 Construction of state space State Space The purpose is to express the state of the vehicle. The selection of the state space will directly affect the agent's understanding of the environment. For the unsignaled intersection environment, the present invention combines the PPO-LSTM algorithm and constructs a state space that matches the algorithm characteristics, so as to give full play to the algorithm's advantages in timing processing problems.

[0026] The state space of the present invention is mainly composed of the following four parts: (1) Vehicle information: vehicle speed , acceleration , facing angle .

[0027] (2) Traffic participating vehicle information: the relative distance between the traffic participating vehicles and the vehicle itself ,angle ,speed , acceleration and the heading angle of the participating vehicles .in, It is the serial number of the vehicles participating in the traffic.

[0028] The geodetic coordinates are established with the center of the vehicle as the origin, the east-west direction of the intersection as the x-axis, and the north-south direction as the y-axis. The angle between the line connecting the center of the vehicle and the center of the participating vehicles and the y-axis is the angle The angle between the speed direction of the traffic participating vehicle and the y-axis is the heading angle of the traffic participating vehicle. .

[0029] (3) Traffic intersection structure and mission target point information: the distance from the center of the intersection to the vehicle ,angle , the distance from the mission target point to the vehicle ,angle .

[0030] The geodetic coordinates are established with the center of the intersection as the origin, the east-west direction as the x-axis, and the north-south direction as the y-axis. The angle between the line connecting the center point of the vehicle and the center point of the intersection and the y-axis is the angle , the angle between the line connecting the center point of the vehicle and the mission target point and the y-axis is the angle .

[0031] (4) Historical information: All state information except historical information in , that is, the information in (1), (2), and (3) corresponding to the time t-1 to tn. In order to distinguish it from the state space, Denoted as (Recommended here 5) is used to record the historical status of vehicles participating in traffic, so as to facilitate the intelligent agent to accurately predict the trajectory of vehicles participating in traffic.

[0032] The final state space is: .

[0033] 2.2 Construction of action space Action Space The agent is in state The action space of the agent is defined as throttle, steering wheel angle and brake, and its range is [0,1], [-1,1], [0,1] respectively. Each time, the three action instructions of the agent can be arbitrarily selected within the above three ranges. In actual vehicle control, the actual output is the maximum output of the vehicle in this capability multiplied by the action value given by the model, for example: actual output throttle signal = maximum throttle output value of the vehicle * agent output action. In the actual vehicle, the throttle signal is converted into the torque output of the engine through the ECU (engine control unit), the steering signal is executed by the electric power steering system (EPS) or the steering gear, and the brake signal is controlled by the brake control unit (BCU) and acts on the hydraulic or electronic brake. The final action space is: .

[0034] 2.3 Construction of reward function Reward Function There are usually two construction forms. The first is to define , the reward of this construction method at the current time step depends only on the current state, that is, the vehicle is rewarded when it is in a safe interaction position, and punished when it is in a dangerous interaction position. The second reward construction method is to define , the reward of this construction method at the current time step depends not only on the current state, but also on the response action taken by the agent in the current state. Since the environment in which the agent is located in the mouth with poor signal is complex and changeable, the second definition method will inevitably fail to cover all situations. Therefore, the present invention adopts the first reward construction method. The quality of the reward function will directly affect the agent's judgment on whether the current action is correct or not. Therefore, formulating a reward function evaluation system that meets the scenario of the present invention is one of the key issues of the present invention. The reward function formulated by the present invention mainly includes the following parts: 2.3.1 One-time Reward Task completion reward: A one-time reward is given when the agent completes a task, to encourage the agent to complete the task.

[0035]

[0036] in, To adjust the parameters, adjust according to the actual task, usually take a larger positive number. Take 500~1000.

[0037] Collision penalty: When an agent collides in an intersection, a one-time large penalty is given to avoid collisions.

[0038]

[0039] in, To adjust the parameters, adjust according to the actual task, usually take a larger negative number. Take -500~-1000.

[0040] 2.3.2 Progressive Rewards (1) Set the progressive reward of the agent's distance from the target point. The specific calculation process is as follows, assuming The position of the agent at the moment is , , The position of the agent at time , , the location of the mission target point is , , first calculate the distance from the agent to the target position at the initial moment , and the distance from the agent to the target position at any time , and its calculation formula is as follows:

[0041]

[0042] Then The distance at the moment is normalized, and the normalized distance is recorded as , and its calculation formula is as follows:

[0043] Finally, the calculation formula of the progressive distance reward value is obtained: ;

[0044] in, , is the weight coefficient, which is determined based on the task scenario, the time required to complete the task, and the weight adjustment between the various rewards. The value may be different in different experimental scenarios. is the value of the current time step, is the normalized distance; in this formula, increasing The value of can encourage the agent to complete the task efficiently, but if Too large a value may cause the vehicle to behave aggressively. If the value of exceeds a certain level, the progressive reward weight will exceed the collision reward weight, causing the agent to ignore the collision risk interaction and increase This phenomenon will be weakened, so the balance , The value between is crucial. After testing and debugging, When the duration is 50ms, , Satisfaction between More suitable.

[0045] (2) Progressive penalty for collision risk In existing algorithms, TTC (Time to Collision) is often used to measure the risk of progressive collision. TTC refers to the time required for a collision between the vehicle and the object in front (usually other vehicles, pedestrians or obstacles) without taking any avoidance measures. Its calculation formula is: ; in, is the current distance between the ego vehicle and the target object. is the relative speed between the ego vehicle and the target object. Usually, it is the speed difference between the ego vehicle and the target object. In the intersection problem, the collision point is usually predicted first. is the distance from the vehicle to the collision point, is the speed of the vehicle, but since this method is too simple to predict the collision point, the collision point is usually predicted by the intersection of the current speed tangents of each vehicle, which may not be accurately predicted in the complex environment interaction of the intersection. Therefore, a new method is proposed in the present invention, that is, risk assessment is performed using the relative position of the vehicle and the relative position change rate. The specific method is as follows: The first step is to determine the vehicle's location and the location of the participating vehicles Calculate the relative position vector , then the vehicle speed ( , ) and the speed of participating vehicles in the traffic ( , ) Calculate the relative velocity vector , the relative speed vector and the relative distance position vector are used to calculate the relative speed of the vehicle along the relative position direction , the calculation process is as follows Figure 3 As shown, the calculation formula is as follows: ; Finally calculate the estimated collision time (Collision time is calculated by the following formula, which is used to represent the time from a certain moment when the vehicle remains in the same state until it collides with other vehicles. Note that in order to avoid the denominator being equal to 0, the speed Under the condition of ; ; And according to The value of divides the vehicle position into low-risk area, medium-risk area, high-risk area and no-risk area, and punishes the agent by dividing the risk area. The specific division is as follows: ; The reward expression here is as follows: ; Here, , , , To adjust the parameters, adjust according to the actual task, usually take a negative number, and satisfy .

[0046] Recommended values ​​here: , , , .

[0047] 2.3.3 Reasonable speed reward and stability reward: This part of the reward is mainly to improve the comfort and efficiency of smart driving vehicles. In order to encourage vehicles to pass through intersections quickly, a small reward is given when the speed is set at 3-30km / h, and a certain penalty is imposed when the speed is greater than 40. At the same time, to ensure riding comfort, a certain penalty is imposed on the vehicle when the absolute value of the vehicle's acceleration and the absolute value of the turning angle change rate are greater than 0.3.

[0048] The reward expression here is: When the vehicle speed is 3-30 hour: ; When the speed exceeds 40 hour: ; Here, , To adjust the parameters, adjust according to the actual task, is a positive number, Is a negative number.

[0049] When the absolute value of acceleration or the rate of change of the angle is greater than 0.3: ;

[0050] Here, To adjust the parameters, adjust according to the actual task, The recommended values ​​are: j=2, k=-10, l=-8.

[0051] Finally get The reward function expression at the moment: ; 3. Constructing a decision model and evaluation model based on LSTM neural network In order to improve the decision-making level of the intelligent agent in the dynamically changing unsignaled intersection scene and overcome the limitations of the traditional MLP neural network in processing complex time-series tasks, the present invention constructs an Actor-Critic architecture based on the LSTM neural network, in which the policy network (Actor network) and the value network (Critic network) both introduce the LSTM structure to process the time-series information in the state space. This design enables the intelligent agent to more effectively capture the dependency between the historical state and the current state, so as to make more reasonable decisions in a dynamic environment.

[0052] 3.1 Data Input State input: input the state information of the previous step into the model 3.2 LSTM layer design The role of the LSTM layer is to extract the temporal features of the input state sequence and capture the dynamic relationship between states. It outputs the hidden state as the input of the subsequent network.

[0053] LSTM parameter configuration: Hidden Units: 128

[0054] Number of Layers: 1.

[0055] Dropout: 0.2, used to prevent overfitting.

[0056] 3.3 Critic Network Construction Network structure: The output of the LSTM layer is passed to the fully connected layer to generate the state value function

[0057] Loss function of state value function:

[0058] The loss function of the action value function is: ; in, is the state value function, for the input state , the output is the value of the current state, for The actual state value at the moment; is the action value function, for the input state , take action , the output is the current state - the value obtained by the action; for The actual action value at the moment. E stands for expectation.

[0059] Output: : The evaluation value of the status.

[0060] 3.4 Actor Network Construction Network structure: The output of the LSTM layer is mapped to the action space through a fully connected layer (FC), and the output is the probability distribution of the action or the specific action value.

[0061] Action selection: The model output is the mean and variance of the action, and Gaussian distribution sampling is used to obtain the final action.

[0062] Among them, the specific network model parameters are shown in Table 1.

[0063] Table 1 Network model parameters

[0064] Here is the difference between batch size and n_step. n_step indicates the number of samples of the model. For example, if n_step is 512, it means that 512 time steps of data are taken from the environment. Batch size indicates the number of samples used for each optimization, that is, the data used for parameter update is not all data 512, but the number of samples in the data, that is, the number of batch sizes, which is 128. n_epochs indicates how many rounds of optimization are performed using the data sampled each time. Total_step indicates the total number of training steps, which is 1*10^6 here.

[0065] 4. Parameter update based on PPO algorithm 4.1 Value Function In order to obtain a state-value function and a strategy-value function with high evaluation accuracy, the present invention takes the state-value function as an example and discusses the process of obtaining this function.

[0066] The first step is to define the loss function of the state value function. The loss function is a function that measures the deviation between the output value of the value network and the actual value. Our goal is to minimize the loss function and ultimately achieve a situation where the value prediction is consistent with the actual value or the difference is within an acceptable range. The loss function is defined as follows:

[0067] in, is the state value function, for the input state , the output is the value of the current state, The actual value calculated by the generalized advantage estimation (GAE) method can be understood as the model being The actual value calculated by the GAE method at each moment. The calculation process can only be performed after the model completes a round of tasks. The calculation method is to record the action, state, reward value and other information of each time step, and then use the GAE method to calculate it. The specific calculation process is as follows: First, calculate the TD error :

[0068] in, yes The actual reward at the moment, in order to facilitate readers to understand the formula, we can Disassembly:

[0069] in, It can be understood as the calculated value of the reward obtained by the model at the current moment. Type In the calculation formula, we can use It is understood as the difference between the model's calculated reward at the current moment and the actual reward.

[0070] Step 2: Calculate the value advantage function , and its calculation formula is as follows: ; , , They are time, time, time, Moment Error. Usually the value is the training step length n_step, which refers to the number of times the algorithm collects a set of states, actions, rewards, etc. from the environment and then uses these data to update the model parameters. For example, n_step=2048, which means that each training uses the data collected in 2048 time steps for multiple optimizations. The number of optimizations depends on n_epochs. If n_epochs is 10, it means that the 2048 data will be used for 10 training optimizations.

[0071] The PPO algorithm uses the Monte Carlo method to calculate this process. is the attenuation factor, parameter It is the balance factor of GAE, usually between 0 and 1 (such as 0.95), which controls the balance between bias and variance.

[0072] Step 3: Calculate :

[0073] In getting Then, we bring it into the loss function to calculate the value of the loss function. According to the calculated loss function, we update the parameters of the value network by the gradient descent method. θ The specific steps are as follows: (1) For all the loss functions θ Find the partial derivative and find the rate of change of the loss function on each parameter .

[0074] (2) Use the required gradient to update the parameters. The update formula is:

[0075] in, is the learning rate, which controls the step size of each update. is the parameter of the current network. So far we have obtained a complete description of the state value The calculation process of the strategy value function The calculation method of is exactly the same as the above process, and no further explanation is given here.

[0076] 4.2 Policy Function First, let’s introduce the strategy function. The PPO algorithm uses a probability-based strategy, that is, given a state , the policy network calculates each possible action The final policy function will output the probability of the agent in The probability distribution of taking each action in a state. , the strategy output of PPO is:

[0077] In order to adjust the parameters of the policy function To update, first define the strategy advantage function , which is used to measure the agent's Take action The calculation formula is as follows:

[0078] Next, we define the importance sampling ratio , this function is used to measure the similarity between the new and old strategies, and its calculation formula is as follows:

[0079] In the formula, the numerator is State selection action The probability of State selection action probability.

[0080] Finally, the loss function is defined as follows:

[0081] Among them, clip is the clipping function, Represents the clipping threshold, which is used to limit the magnitude of the strategy update; Usually 0.1 or 0.2 is used; the usage here is when The value is in [ , ] takes the original value, when Less than The output result is ,when Greater than The output result is The final result is and This definition of the loss function limits the amplitude of parameter updates by introducing a clipping function. Specifically, when the clipping function does not work, the policy gradient is directly based on the loss function. With A( , ) is used for calculation. When the clipping function takes effect, it means that the difference between the new and old strategies is too large, and the loss function becomes a loss function without The constant of , resulting in a gradient of 0, limits the gradient update amplitude and avoids drastic policy updates.

[0082] Specific update of parameters in the strategy function: (1) Parameters of all policy functions in the loss function Find the partial derivative and find the rate of change of the loss function on each parameter ; (2) Use the required gradient to update the parameters. The update formula is:

[0083] The training ends when the algorithm's loss function finally converges and falls below the convergence standard. Combining the model's performance and the convergence status and value of the reward function, we can determine whether the model training effect meets the requirements. If the effect is not good, we can adjust the parameters to achieve the ideal effect.

[0084] The present invention improves the reinforcement learning algorithm and embeds the long short-term memory network (LSTM) into the policy network architecture of the PPO algorithm, thereby achieving the purpose of making more efficient decisions at unsignalized intersections using the historical status data of the weekly vehicles; by constructing a Markov decision model for unsignalized intersections that matches the algorithm, the policy model's ability to understand the task is improved; by adopting online reinforcement learning (real-time data collection, learning and updating during the interaction with the environment), the obtained model has self-learning ability, which can continuously improve the traffic efficiency during the interaction process, and enhance the exploration and risk tolerance of the decision model for unsignalized intersection scenarios; at the same time, it can improve the model convergence speed, save storage and computing resources, and reduce the model computing overhead. Ultimately, the purpose of improving the vehicle traffic strategy at the intersection and reducing the conservatism of the decision-making process is achieved.

[0085] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A method for intelligent vehicle decision making at an unsignalized intersection, characterized in that: include: Constructing an intelligent vehicle decision model for an unsignalized intersection, the decision model comprising: a strategy network and a value network; Wherein, the strategy network and the value network are both composed of LSTM neural networks and fully connected layers; The information of the ego vehicle from time tn to time t, the information of the participating vehicles in the traffic, the relative position information of the traffic intersection and the ego vehicle, and the relative position information of the target task point and the ego vehicle form the state parameters as the input variables of the strategy network, and the throttle control signal, steering wheel angle control signal and brake control signal corresponding to the state parameters at time t form the action parameters as the output variables of the strategy network; The state parameter and the action parameter are input into the value network, and the value network outputs the value corresponding to the state parameter and the action parameter; The PPO algorithm is used to construct the loss functions of the policy network and the value network respectively, and the decision model is trained. The parameters of the policy network and the value network are optimized respectively with the minimum loss function as the optimization goal to obtain the optimal decision model; When the vehicle travels to an unsignaled intersection, state parameters are acquired in real time and input into the optimal decision model, and the optimal decision model outputs an optimal throttle control signal, a steering wheel angle control signal, and a brake control signal.

2. The intelligent vehicle decision-making method for an unsignalized intersection according to claim 1, characterized in that: The loss function of the policy network is: ； in, is the clipping function, is the importance sampling ratio, are policy network parameters, is the strategy advantage function, and They are The state at the moment and the corresponding action, represents the clipping threshold; Expressing expectation.

3. The intelligent vehicle decision-making method for an unsignalized intersection according to claim 2 is characterized in that: The loss function of the value network is: ； in, is the state value function, for the input state , the output is the value of the current state; for The actual value of the moment.

4. The intelligent vehicle decision-making method for an unsignalized intersection according to claim 3 is characterized in that: The actual value is calculated as follows: ； in, is the value advantage function; ； , , They are time, time, time, TD error at the moment; The calculation formula of TD error at time is: ； In the formula, for The actual reward of the moment, The corresponding state of the value network output The value of is the attenuation factor.

5. The intelligent vehicle decision-making method for an unsignalized intersection according to claim 4 is characterized in that: The actual reward is calculated by the following reward function: ； in, The reward value for completing the task. is the collision penalty value, is the progressive distance reward value, is the progressive penalty value of collision risk, is the reasonable speed reward value, is the stability reward value.

6. The intelligent vehicle decision-making method at an unsignalized intersection according to claim 5, characterized in that: The calculation formula of the progressive reward is: ； in, is the value of the current time step, is the normalized distance, , , is the weight coefficient; is the distance from the vehicle to the target position at the initial moment, for The distance from the vehicle to the target location at any moment.

7. The intelligent vehicle decision-making method at an unsignalized intersection according to claim 5 or 6, characterized in that: The method for calculating the progressive penalty value of the collision risk comprises the following steps: Step 1: According to the vehicle position ( , ) and the positions of participating vehicles in the traffic ( , ) Calculate the relative position vector ; According to the vehicle speed ( , ) and the speed of participating vehicles in the traffic ( , ) Calculate the relative velocity vector ; Step 2: Calculate the relative speed of the vehicle along the relative position direction : ； Step 3: Calculate the estimated collision time : ； Step 4: Determine the progressive penalty value of collision risk based on the expected collision time: ； in, , , , is the adjustment parameter and satisfies .

8. The intelligent vehicle decision-making method for an unsignalized intersection according to claim 7, characterized in that: The LSTM neural network used by the strategy network and the value network both has 1 layer and 128 hidden units.

9. The intelligent vehicle decision-making method at an unsignalized intersection according to claim 8, characterized in that: The number of neurons in the fully connected layer used by the strategy network and the value network is 64.

10. The intelligent vehicle decision-making method at an unsignalized intersection according to claim 9, characterized in that: The activation function used by the fully connected layer of the strategy network is Tanh, and the activation function used by the fully connected layer of the value network is ReLU.

Images