An individualized automatic lane-changing and speed control driving collaborative decision-making method

Abstract

The application relates to a personalized automatic lane-changing and speed control driving cooperative decision method, which comprises the following steps: S1, constructing a personalized decision hierarchical framework; S2, building a lane-changing decision training network, determining the optimization target of the lane-changing decision, introducing an action shielding mechanism to eliminate the safety hazards of dangerous actions, setting the upper state space, the action space and the lane-changing reward function required for training, and giving the lane-changing reward function personalized weights; S3, building a following decision training network, determining the optimization target of the following decision, quantifying the optimization target into a reward coefficient, setting the lower state space, the action space and the following reward function required for training, and giving the following reward function personalized weights; S4, adopting a cooperative decision method and coupling training of the lane-changing decision training network and the following decision training network, and building a style reward priority experience playback to enrich the experience playback priority measurement. Compared with the prior art, the application has the advantages of improving the safety and driving efficiency of a vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent driving, in particular to a personalized automatic lane changing and speed control driving cooperative decision-making method. BACKGROUND

[0002] With the rapid development of artificial intelligence, learning-based methods have been widely applied in intelligent driving decision control systems, making driving behavior decision-making have better generalization performance. Among them, deep reinforcement learning combines the advantages of deep learning representation and the advantages of reinforcement learning reasoning, providing an intelligent solution for vehicle behavior decision-making.

[0003] Automatic driving decision control includes lane changing decision and following control. Lane changing decision examines the driving relationship between vehicles on multiple lanes, while following control mainly focuses on the driving conditions of vehicles on the same lane. Traditionally, in the automatic driving decision control system, the study of lane changing decision and following control is carried out as a separate work, ignoring the potential cooperation between the two.

[0004] In the field of automatic driving, there is a strong connection between lane changing and following during driving, and lane changing behavior changes the trajectory and speed control strategy of the vehicle, making the complete lane changing process involve the lateral and longitudinal motion of the vehicle. In addition, factors such as following distance and speed can directly affect lane changing decision, and if the following condition is poor, the vehicle may change lanes to seek a smoother route. Therefore, studying the cooperative relationship between lane changing and following helps to find the optimal action in complex traffic environments, complete more accurate, efficient and safe driving behavior decision control, and thus better achieve driving intent to improve the safety and efficiency of vehicle driving.

[0005] In recent years in the field of deep reinforcement learning, although the internal relationship between lane changing and following is considered, the difference in driving habits between drivers of different driving styles is ignored, which may lead to large decision bias. With the development of intelligent connected vehicles and advanced driving assistance systems (ADAS), driving style as a comprehensive evaluation index of driving has important significance for the intelligence and safety of human-machine interaction. Different driving styles will affect the decision-making behavior of drivers, thereby affecting the performance of autonomous driving vehicles. In automatic driving decision control, considering the influence of driving style can make the decision-making method more consistent with the driving habits of drivers, improving the safety and comfort of driving.

[0006] In order to solve the above technical problems in the field, researchers need to study the cooperative relationship between lane changing behavior and following control from the perspective of driving decision control, and optimize and adjust based on different driving styles to make more reasonable control decisions, and propose a personalized automatic lane changing and speed control cooperative decision-making method. SUMMARY

[0007] The purpose of the present application is to provide a personalized automatic lane changing and speed control driving collaborative decision-making method.

[0008] The purpose of the present application can be achieved by the following technical solutions:

[0009] A personalized automatic lane changing and speed control driving collaborative decision-making method, comprising:

[0010] Step S1: constructing a personalized decision-making hierarchical framework, wherein the personalized decision-making hierarchical framework is based on feature parameters obtained through decision result simulation to provide a shared state space;

[0011] Step S2: building a lane changing decision-making training network, determining the optimization target of lane changing decision-making, introducing an action shielding mechanism to eliminate the safety hazards of dangerous actions, setting the upper state space, upper action space and lane changing reward function required for training, and giving personalized weights to the lane changing reward function, wherein the upper state space is determined based on the shared state space;

[0012] Step S3: building a following decision-making training network, determining the optimization target of following decision-making, quantifying the optimization target into a reward coefficient, and setting the lower state space, lower action space and following reward function required for training, and giving personalized weights to the following reward function, wherein the lower state space is determined based on the shared state space;

[0013] Step S4: using a collaborative decision-making method and coupling training of the lane changing decision-making training network and the following decision-making training network, and building a style reward priority experience playback to enrich the experience playback priority measurement.

[0014] The feature parameters include vehicle heading angle, vehicle headway, collision time, acceleration impact degree and lane changing duration.

[0015] In the lane changing decision-making training network, the optimization target of lane changing decision-making is:

[0016] Traget lc =max(v e )+min(tlc)+min(dlc)

[0017] Wherein: Traget lc is the optimization target of lane changing decision-making, v e is the driving speed of the vehicle, tlc is the total number of lane changes, and dlc is the number of dangerous lane changes.

[0018] The action shielding mechanism includes:

[0019] After changing to the target lane, the deceleration of the vehicle and the rear vehicle cannot be less than the minimum safe deceleration to ensure that the vehicle has sufficient deceleration to avoid a collision;

[0020] The distance between vehicles ensures a safe distance:

[0021]

[0022] wherein d max is the maximum distance between the vehicle and the front vehicle, d safe is the safe following distance at the current speed of the vehicle, t b is the reaction time of the driver, a safe is the minimum safe deceleration of the road.

[0023] The upper state space is:

[0024] LC_State = {v e , a e , {v pi , a pi , h i , pd i , pv i}}

[0025] wherein LC_State is the upper state space, a e is the acceleration of the vehicle, v pi is the speed of the front vehicle in the ith lane, a pi is the acceleration of the front vehicle in the ith lane, h i is the headway of each vehicle in the ith lane, pd i is the position advantage of each vehicle in the ith lane, pv i is the speed advantage of each vehicle in the ith lane.

[0026] The upper action space is:

[0027] LC_Action = {0, 1, 2}

[0028] wherein LC_Action is the upper action space, 0 represents maintaining the current lane, 1 represents changing lanes to the left, and 2 represents changing lanes to the right.

[0029] The lane changing reward function is:

[0030] LC_Reward = α*v e + γ*lc + ω*c

[0031] wherein LC_Reward is the lane changing reward function, lc is whether a lane changing behavior occurs, c is the number of collisions, and α, γ, ω are weights.

[0032] The position advantage of the vehicle in the ith lane is:

[0033]

[0034] Wherein: y e is the longitudinal distance between the host vehicle and the rear vehicle in the ith lane when the host vehicle decides to change lanes, y ri is the longitudinal distance between the host vehicle and the rear vehicle in the ith lane when the host vehicle decides not to change lanes, v ri is the speed of the rear vehicle in the ith lane, t b is the reaction time of the driver;

[0035] The speed advantage of each vehicle in the ith lane is:

[0036]

[0037] Wherein: LC prob is the feasibility of the vehicle's lane changing behavior, v pi is the speed of the front vehicle in the ith lane, t 1i is the time required to accelerate to the maximum speed limit of the road, t 2i is the time to maintain the maximum speed limit, t 3i is the time required to decelerate to the speed of the front vehicle.

[0038] The car-following decision training network, the optimization target of the car-following decision is:

[0039]

[0040] Wherein: Traget cf is the optimization target of the car-following decision, v e is the driving speed of the host vehicle, a c is the acceleration of the front vehicle, and c is the number of collisions occurring during car-following.

[0041] The lower state space is:

[0042] F_State = {v e ,v p ,a p ,h}

[0043] Wherein: F_State is, v p is the speed of the vehicle in front of the current lane, a p is the acceleration of the vehicle in front of the current lane, and h is the distance between the leading vehicle and the front vehicle when car-following in the current lane;

[0044] The lower action space is:

[0045] CF_Action = {a e ∈ [a dec ,a acc}]

[0046] wherein: CF_Action is the lower action space, a e is the acceleration of the ego vehicle, a dec is the maximum deceleration of the ego vehicle, a acc is the maximum acceleration of the ego vehicle;

[0047] The car-following reward function is:

[0048]

[0049] wherein: CF_Reward is the car-following reward function, β, λ, μ are weights, is the mean square of the vehicle acceleration.

[0050] The style reward priority experience replay divides the experience priority calculation into a TD error index and an incentive index, wherein the incentive index is composed of R_Advantage and Q_Advantage, and is used to evaluate different driving styles for different rewards and speed exploration preferences, and improve the convergence of the algorithm. Based on the reward expectation and target preference under different styles, a personalized weight coefficient is given to the sample priority. For different transitions (s t ,a t ,r t ,s t+1 ),, the sample priority P(t) is defined as:

[0051] P(t) = ρ*P TD (t) + (1-ρ)*P I (t) + ε

[0052]

[0053] wherein: ρ is the weight of the TD error index, P TD (t) is the TD error index, P I (t) is the incentive index, and ε is a trace constant, is the average TD error, η1, η2 are constants, rt is the short-term reward, and Q(s t ,a t ) is the long-term reward;

[0054] The average TD error is:

[0055]

[0056] wherein: Q traget (s t ,a t ) is a TD target value calculated according to the reward obtained from the environment, and Q(s t ,a tθ j ) is the Action-value function value Q estimated by the Critic network j in the car-following decision training network.

[0057] The car-following decision training network is built based on the SA3C algorithm, and the lane-changing decision training network is built based on the D3QN algorithm;

[0058] The coupling training includes:

[0059] Step S4-1: Initialize the neural network parameters for the D3QN and SA3C algorithms, including:

[0060] Random initial network parameters: θ and θ′←θ; set the maximum capacity U of the replay buffer d and the number of training batches B d ,

[0061] Initialize the parameters of SA3C, including the Actor network parameters θ μ 、Critic network parameter φ i (i=1,2,3) and target network parameters θ μ′ ←θ μ ,φ′ i ←φ i ; Set the maximum capacity of the playback buffer U r and the number of training batches B r ;

[0062] Step S4-2: In each training round, select and execute actions according to the current strategy, and accept the initial observation state s from the simulation environment:

[0063] (1) Process the Q value of the dangerous action according to the current state, select and execute the action

[0064] If the action corresponding to the maximum value of the current Q(s,a|θ) is the safest, then the current action is selected. Otherwise, with a certain probability Randomly select an action

[0065] (2) Use exploration noise ζ to select the execution action

[0066] Execute an action and By calculating the reward function and observing the reward r t and the next state s t+1 ,

[0067] (3) Transfer and are stored in the replay buffer D and the following buffer R, respectively;

[0068] Step S4-3, using the SIPER to sample a batch of transition data from and , and calculate the D3QN target value y j = r j + γQ(s', argmaxQ(s', a'|theta); theta'). j = r j + γQ(mst i＝1,2,3 Q(s', a'|phi i )- alpha log mu(s'|theta μ )).

[0069] Step S4-4, update the network parameters of the D3QN algorithm through the gradient descent optimization algorithm:

[0070]

[0071] Step S4-5, update the Criti c network parameters of the SA3C algorithm by minimizing the loss L:

[0072]

[0073] Step S4-6, update the Actor network parameters of the SA3C algorithm using the sampled policy gradient:

[0074]

[0075] Step S4-7, update the priority weights of the transition and in D and R and replace the target neural network parameters theta ′ of the D3QN algorithm and the SA3C target neural network parameters theta μ , phi i every certain period of time. ′ .

[0076] Compared with the prior art, the present application has the following beneficial effects:

[0077] 1. The present application proposes personalized decision planning with customized reward function weights to adapt to the needs and preferences of different drivers.

[0078] 2. The present application uses hierarchical reinforcement learning to realize collaborative decision making, which is composed of a D3QN planner with dangerous lane changing behavior and an improved SA3C car following planner with three evaluation networks to improve stability.

[0079] 3、The application considers the reward preference and strategic vision under different driving styles, and proposes a style incentive priority experience replay mechanism (SIPER), which integrates immediate rewards and long-term returns into experience priority calculation and gives different sample priority weight coefficients.

[0080] 4、The application divides the lane advantage into speed advantage and position advantage, and incorporates the shared state space to reflect the interaction between the agent and the surrounding vehicles. BRIEF DESCRIPTION OF DRAWINGS

[0081] Figure 1 It is a schematic diagram of the hierarchical decision-making framework process;

[0082] Figure 2 It is a schematic diagram of the decision-making process in a complex traffic environment;

[0083] Figure 3 It is a schematic diagram of the hierarchical collaborative decision-making framework structure;

[0084] Figure 4 It is a schematic diagram of the main steps of the application. DETAILED DESCRIPTION

[0085] The application will be described in detail below in conjunction with the drawings and specific embodiments. The embodiments are implemented on the basis of the technical solutions of the application, and detailed implementation methods and specific operation processes are given, but the protection scope of the application is not limited to the following embodiments.

[0086] Overall, the application takes automatic lane changing and speed control of different driving styles as the research object, develops a personalized driving decision-making framework based on the driving style perception decision-making method of deep reinforcement learning. In addition, based on the D3QN algorithm and the flexible actor-critic (SA3C) algorithm, a personalized hierarchical decision-making framework for lane changing and following is built, and the coupling training of the two is carried out to achieve the collaborative control effect of following and lane changing. The main steps include driving behavior feature extraction, replacing statistical feature calculation; building a traffic simulation environment, including lane changing decision, following control and multi-vehicle interaction; building a personalized hierarchical decision-making framework for lane changing and following, building actor (Actor) and critic (Critic) networks and target critic networks to form an agent; training the hierarchical driving strategy, obtaining the inheritable network strategy through the interaction between the environment and the agent; processing the trajectory driving data, performing trajectory experiments, and simulating training.

[0087] This method utilizes a D3QN with prioritized experience replay as the core of lane-changing decision-making. It combines the advantage function with the lane-changing benefit function and introduces an action masking mechanism to gradually eliminate the safety hazards of dangerous maneuvers. SA3C optimizes the maximum expected reward and policy entropy, encouraging exploration and improving training stability. Furthermore, by analyzing driver behavioral characteristics and considering the impact of different driving styles to meet personalized driving needs, it can help vehicles make more reasonable lane-changing decisions and follow-up control.

[0088] To make the objectives, technical solutions, and advantages of the present invention more clear, the following further describes the technical solutions of the application in detail with reference to the accompanying drawings. The examples described are only part of the embodiments involved in this patent. All non-innovative examples based on this embodiment by other researchers in this field fall within the scope of protection of this patent.

[0089] A personalized automatic lane change and speed control driving collaborative decision-making method, such as Figure 4 and Figure 3 As shown, including:

[0090] Step S1: Construct a personalized decision-making hierarchical framework, where Figure 1 As shown, the personalized decision hierarchical framework obtains feature parameters based on the decision result simulation to provide a shared state space;

[0091] Specifically, based on the differences in driving decisions among different driving styles, a personalized hierarchical decision-making framework is established. The simulation environment is a lane-changing and following decision-making environment in a multi-lane scenario with mixed traffic flow:

[0092] The driving style characteristics of lateral and longitudinal vehicle motion are extracted from the autonomous driving style, and the influence of human factors is taken into account. According to the existing natural driving data research, the vehicle heading angle, headway, collision time, acceleration impact degree and lane change duration are mainly selected as characteristic parameters.

[0093] In the decision-making goal design, the lane-changing scenario process is as follows: Figure 2 As shown in the figure, during a free lane change, a vehicle considers changing lanes when it is blocked by the vehicle ahead and cannot achieve the desired speed gain in the current lane. At this point, the vehicle considers the driving conditions and safety advantages of adjacent lanes. Furthermore, to prevent frequent and arbitrary lane changes, a penalty mechanism is implemented to keep the number of lane changes within a reasonable range.

[0094] Step S2: Build a lane-changing decision training network, determine the optimization goal for lane-changing decisions, introduce an action shielding mechanism to eliminate safety hazards of dangerous actions, set the upper state space, upper action space, and lane-changing reward function required for training, and assign personalized weights to the lane-changing reward function. The upper state space is determined based on the shared state space.

[0095] Lane changing scenario process is shown as Figure 2 In free lane changing, when the vehicle is blocked by the front vehicle and cannot obtain the expected speed gain in the current lane, the vehicle will consider changing lanes. At this time, the vehicle will weigh the driving conditions and safety advantages and disadvantages of adjacent lanes. At the same time, in order to prevent the vehicle from frequently and randomly changing lanes, the number of lane changes is controlled within a reasonable range through a punishment mechanism, therefore, in the lane changing decision training network, the optimization target of lane changing decision is:

[0096] Traget lc = max(v e )+min(tlc)+min(dlc)

[0097] Wherein: Traget lc is the optimization target of lane changing decision, v e is the driving speed of the vehicle, tlc is the total number of lane changes, and dlc is the number of dangerous lane changes.

[0098] The action shielding mechanism includes:

[0099] After changing lanes to the target lane, the deceleration of the vehicle and the rear vehicle cannot be less than the minimum safe deceleration to ensure that the vehicle has sufficient deceleration to avoid collision, specifically:

[0100]

[0101] Wherein: LC prob is the lane changing feasibility of the vehicle, a′ e is the deceleration of the vehicle after lane changing, a safe is the minimum safe deceleration of the road, a′ en is the deceleration of the vehicle behind the target lane,

[0102] At the same time, in order to pursue higher driving efficiency, the distance between vehicles should not be too large, and the safety distance should also be ensured, and the distance between vehicles ensures the safety distance:

[0103]

[0104] Wherein: d max is the maximum distance between the vehicle and the front vehicle, d safe is the safe following distance at the current speed of the vehicle, and t b is the reaction time of the driver.

[0105] The upper state space is:

[0106] LC_State = {v e , a e , {v pi , a pi , h i , pdi pv i}}

[0107] wherein: LC_State is the upper layer state space, a e is the acceleration of the ego vehicle, v pi is the speed of the front vehicle in the ith lane, a pi is the acceleration of the front vehicle in the ith lane, h i is the headway of each vehicle in the ith lane, pd i is the position advantage of the vehicle in the ith lane, pv i is the speed advantage of each vehicle in the ith lane.

[0108] wherein, in the lane-changing scenario, the distance between the front and rear vehicles in the target vehicle, and the competitive relationship between the rear vehicle in the target lane and the ego vehicle are mainly considered.

[0109] When the ego vehicle decides to change lanes, the aggressive rear vehicle will accelerate to protect its rights and narrow the distance gap with the front vehicle, thereby preventing the host vehicle from changing lanes. The conservative rear vehicle will slow down to avoid a collision and expand the gap, thereby giving way to the vehicle in the ego lane. Therefore, the position advantage is introduced to represent the competitive relationship between vehicles. Specifically, the position advantage of the vehicle in the ith lane is:

[0110]

[0111] wherein: y e is the longitudinal distance between the ego vehicle and the rear vehicle in the ith lane when the ego vehicle decides to change lanes, y ri is the longitudinal distance between the ego vehicle and the rear vehicle in the ith lane when the ego vehicle decides not to change lanes, v ri is the speed of the rear vehicle in the ith lane, t b is the driver's reaction time, which is also used as a safety threshold for the headway, generally 1.5 seconds;

[0112] In the following vehicle scenario, the front vehicle in the current lane is mainly considered. If the front vehicle is aggressive and drives fast, the maximum speed advantage that the ego vehicle can obtain will also be enlarged. If the front vehicle is conservative and drives slowly, the ego vehicle cannot drive too fast, thereby obtaining a lower speed advantage. The speed advantage is calculated under the assumption that the ego vehicle catches up with the front vehicle in the lane as fast as possible. Specifically, the speed advantage of each vehicle in the ith lane is:

[0113]

[0114] wherein: v pi is the speed of the front vehicle in the ith lane, t 1i is the time required to accelerate to the maximum speed limit of the road, t 2i is the time required to maintain the maximum speed limit, t 3i is the time required to decelerate to the speed of the front vehicle.

[0115]

[0116] t 1i and t 3i After the stage, the distance between the car and the front car is:

[0117] s di = (t 1i + t 3i )*v pi + h i -s 1i -s 3i

[0118] When s di is less than 0, the maximum speed that the ego vehicle can reach is less than the maximum speed limit of the road, there is no stage to maintain the maximum speed limit of driving, at this time the acceleration and deceleration process of the ego vehicle is calculated as follows:

[0119]

[0120] Where v′ max is the maximum allowed speed on the lane, t1 ′ i is the time for the car to accelerate to the maximum allowed speed, t3 ′ i is the time from the maximum allowed speed to the speed of the front vehicle.

[0121] The upper action space is:

[0122] LC_Action = {0, 1, 2}

[0123] Where: LC_Action is the upper action space, 0 represents maintaining the current lane, 1 represents changing lanes to the left, and 2 represents changing lanes to the right;

[0124] The lane changing reward function is:

[0125] LC_Reward = a*v e + g*lc + w*c

[0126] Where: LC_Reward is the lane changing reward function, lc is whether the lane changing behavior occurs, c is the number of collisions, and a, g, w are weights.

[0127] Step S3: build a following decision training network, determine the optimization target of the following decision, at the same time, quantize the optimization target to a reward coefficient, and set the lower state space, lower action space and following reward function required for training, and give the following reward function personalized weights, wherein the lower state space is determined based on the shared state space;

[0128] The follow-up decision training network, the optimization goal of follow-up decision is:

[0129]

[0130] Wherein: Traget cf The optimization goal of follow-up decision, v e The driving speed of the vehicle, a c The acceleration of the front vehicle, c is the number of collisions occurring in the follow-up process.

[0131] The lower state space is:

[0132] F_State={v e ,v p ,a p ,h}

[0133] Wherein: F_State is, v p The speed of the vehicle in front of the current lane, a p The acceleration of the vehicle in front of the current lane, h is the distance between the leading vehicle and the front vehicle when following in the current lane;

[0134] The lower action space is:

[0135] CF_Action={a e ∈[a dec ,a acc ]}

[0136] Wherein: CF_Action is the lower action space, a e The acceleration of the vehicle, a dec The maximum deceleration of the vehicle, a acc The maximum acceleration of the vehicle;

[0137] The follow-up reward function is:

[0138]

[0139] Wherein: CF_Reward is the follow-up reward function, β, λ, μ is the weight, The mean square of vehicle acceleration.

[0140] The personalized driving hierarchical coordination decision-making process is as follows Figure 3The agent first acquires the current driving state information of itself and surrounding vehicles, as well as the advantage variables of adjacent lanes, which can represent the driving state of the vehicle itself and the interaction between the agent and the surrounding vehicles, as the shared state space into the decision module. The decision framework is composed of upper and lower planners. Then, the upper planner makes personalized lane-changing decisions according to the required state information and the competitive advantage relative to the surrounding vehicles. When the agent decides to maintain the current lane, the state information of the current lane is transmitted to the car-following planner. The lower planner will provide a continuous speed control strategy that matches the driving style for the next step. Otherwise, it adopts lane-changing behavior and updates the state space of the car-following planner using the state variables of the target lane.

[0141] Step S4: Coupling training of the lane-changing decision training network and the car-following decision training network is carried out using the cooperative decision method, and style reward priority experience replay is built to enrich the experience replay priority metric.

[0142] The style reward priority experience replay divides the experience priority calculation into a TD error indicator and an incentive indicator, where the incentive indicator is composed of R_Advantage and Q_Advantage, which are used to evaluate the different driving styles for different rewards and speed exploration preferences, improving the convergence of the algorithm. Based on the reward expectation and target preference under different styles, a personalized weight coefficient is given to the sample priority. For different transitions (s t ,a t ,r t ,s t+1 ), the sample priority P(t) is defined as:

[0143] P(t) = p * P TD (t) + (1-p) * P I (t) + ε

[0144]

[0145]

[0146] where: p is the weight of the TD error indicator, P TD (t) is the TD error indicator, P I (t) is the incentive indicator, and ε is a trace constant, is the average TD error, η1 and η2 are constants, rt is the short-term reward, and Q(s t ,a t ) is the long-term reward.

[0147] The average TD error is:

[0148]

[0149] where: Q traget (s t ,a t ) is the TD target value calculated from the reward obtained from the environment, Q(s t ,a t ; θ j ) is the Action-value function value Q estimated by the Critic network j in the car-following decision training network.

[0150] Assuming that the experiences in the buffer follow a probability distribution p ψ ~ φ, the priority probability of the transition (s t ,a t ,r t ,s t+1 ) is calculated as follows:

[0151]

[0152] Inheriting the traditional PER and tree experience storage structure, efficient performance level management not only improves the convergence speed, but also changes the sampling experience distribution, introducing potential estimation bias. By assigning sample importance sampling weights (ISW) to correct errors:

[0153]

[0154] where ω t represents the ISW of the transition with priority probability p(t), bs represents the batch size, β represents the importance sampling correction index, which linearly degrades from the initial value β0 to 1 over time. ω t can be applied to the Q-learning update.

[0155] In addition, the car-following decision training network is built based on the SA3C algorithm, and the lane-changing decision training network is built based on the D3QN algorithm;

[0156] Coupled training includes:

[0157] Step S4-1: Initialize the neural network parameters for D3QN and SA3C algorithms, including:

[0158] Randomly initialize network parameters: θ and θ ′ ← θ; Set the maximum capacity U of the replay buffer d and the number of training batches B d ,

[0159] Initialize the parameters of SA3C, including the Actor network parameters θ μ , the Critic network parameters φ i (i = 1, 2, 3) and the target network parameters​ Setting the maximum capacity U of the replay buffer r and the number of training batches B r ;

[0160] Step S4-2, in each training round, an action is selected and performed according to the current policy, and an initial observation state s is received from the simulation environment:

[0161] (1) Process the Q value of the dangerous action according to the current state, select and perform the action

[0162] If the maximum value of the current Q(s, a|θ) corresponds to the safest action, the current action is selected, otherwise, with a certain probability Randomly select an action

[0163] (2) Select the executed action using exploration noise ζ

[0164] Perform the action and Calculate the reward function and observe the reward r t and the next state s t+1 ,

[0165] (3) Store the transition and in the replay lane-changing decision buffer D and the following decision buffer R, respectively;

[0166] Step S4-3, using SIPER to sample a batch of transition data from and , and calculate the D3QN target value y j = r j + γQ(s', argmaxQ(s', a'|θ); θ'). Calculate the SA3C target value y j = r j + γQ(mst i＝1,2,3 Q(s', a'|φ i )- αlogμ(s'|θ μ )).

[0167] Step S4-4, update the network parameters of the D3QN algorithm through the gradient descent optimization algorithm:

[0168]

[0169] Step S4-5, update the Critic network parameters of the SA3C algorithm by minimizing the loss L:

[0170]

[0171] Step S4-6, update the Actor network parameters of the SA3C algorithm using the sampled policy gradient:

[0172]

[0173] Step S4-7, update the transition and Priority weights in D and R and replace the target neural network parameters θ of the D3QN algorithm every certain period of time ′ and the SA3C target neural network parameters θ μ , φ i ′ .

[0174] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, etc.

Claims

1. A personalized automatic lane change and speed control driving collaborative decision-making method, characterized in that: include: Step S1: constructing a personalized decision-making hierarchical framework, wherein the personalized decision-making hierarchical framework obtains characteristic parameters based on decision result simulation to provide a shared state space; Step S2: Building a lane-changing decision training network, determining the optimization goal of lane-changing decisions, introducing an action shielding mechanism to eliminate safety hazards of dangerous actions, setting the upper state space, upper action space, and lane-changing reward function required for training, and assigning personalized weights to the lane-changing reward function, wherein the upper state space is determined based on the shared state space; Step S3: Building a car-following decision-making training network, determining an optimization target for the car-following decision-making, quantifying the optimization target as a reward coefficient, setting the lower-level state space, lower-level action space, and car-following reward function required for training, and assigning a personalized weight to the car-following reward function, wherein the lower-level state space is determined based on the shared state space; Step S4: Adopting a collaborative decision-making approach and performing coupled training of the lane-changing decision training network and the car-following decision training network, and building a style reward-prioritized experience replay to enrich the experience replay priority metric; The style reward priority experience playback divides the experience priority calculation into TD error index and incentive index, where the incentive index consists of R_Advantage and Q_Advantage, which are used to evaluate the different reward and speed exploration preferences of different driving styles, improve the convergence of the algorithm, and give the sample priority personalized weight coefficient based on the reward expectation and target preference under different styles. , its sample priority Defined as: in: is the weight of the TD error indicator, is the TD error indicator, As an incentive indicator, is a trace constant, is the average TD error, 、 is a constant, rt For short-term rewards, For long-term rewards; The average TD error is: in: is the TD target value calculated based on the reward obtained from the environment, Training the Critic network in the car-following decision-making network j The estimated action-value function value Q.

2. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 1, characterized in that: The characteristic parameters include vehicle heading angle, headway, collision time, acceleration impact degree and lane change duration.

3. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 1, characterized in that: In the lane-changing decision training network, the optimization goal of the lane-changing decision is: in: is the optimization goal of lane-changing decision, is the vehicle's speed, is the total number of lane changes, The number of dangerous lane changes.

4. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 3, characterized in that: The upper state space is: in: is the upper state space, is the acceleration of the vehicle, For the i Speed ​​of the vehicle ahead in the lane, For the i Acceleration of the vehicle ahead in the lane, For the first i The headway time of vehicles in each lane, For the i The position advantage of lane vehicles, For the i The speed advantage of each vehicle in the lane; The upper action space is: in: is the upper action space, 0 means keeping the current lane, 1 means changing lanes to the left, and 2 means changing lanes to the right; The lane-changing reward function is: in: is the lane-changing reward function, Whether a lane change occurs, c is the number of collisions, 、 、 is the weight.

5. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 4, characterized in that: The position advantage of the vehicle in lane i is: in: When the vehicle decides to change lanes, the vehicle and i The longitudinal distance of the vehicle behind the lane, When the vehicle decides not to change lanes, the vehicle and i The longitudinal distance of the vehicle behind the lane, is the speed of the car behind in lane i, Driver reaction time; The speed advantage of each vehicle in lane i is: in: The feasibility of vehicle lane change behavior, For the i The speed of the car ahead in the lane, The time required to accelerate to the maximum speed limit of the road, To maintain the maximum speed limit, The time required to decelerate to the speed of the vehicle ahead.

6. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 1, characterized in that: The optimization goal of the car-following decision training network is: in: To optimize the car-following decision, is the vehicle's speed, a c is the acceleration of the preceding vehicle, c is the number of collisions that occurred during the following process.

7. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 6, characterized in that: The lower state space is: in: is the lower state space, is the speed of the vehicle ahead in the current lane, is the acceleration of the vehicle in front lane, h The distance between the leading vehicle and the vehicle in front when following in the current lane; The lower action space is: in: is the lower action space, a e is the acceleration of the vehicle, is the maximum deceleration of the vehicle, is the maximum acceleration of the vehicle; The following reward function is: in: is the following reward function, 、 、 is the weight, is the vehicle's speed, is the vehicle acceleration impact, d is the distance between the vehicle and the vehicle in front.

8. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 1, characterized in that: The car-following decision training network is built based on the SA3C algorithm, and the lane-changing decision training network is built based on the D3QN algorithm; The coupling training includes: Step S4-1: Initialize the neural network parameters for the D3QN and SA3C algorithms, including: Random initial network parameters: ; Set the maximum capacity of the replay buffer and the number of training batches , Initialize SA3C parameters, including Actor network parameters , Critic network parameters and target network parameters ; Set the maximum capacity of the playback buffer and the number of training batches ; Step S4-2: In each training round, select and execute actions according to the current strategy and receive the initial observation state from the simulation environment. s ,include: Handle dangerous actions according to the current state Value, select and perform actions If the current The action corresponding to the maximum value of is the safest, then the current action is selected, otherwise, with a certain probability Randomly select an action , Use Exploration Noise Select the action to perform Execute an action and , by calculating the reward function and observing the reward and the next state , will transfer and They are stored in the playback lane-changing decision buffer D and the car-following decision buffer R respectively; Step S4-3, using style incentive priority experience playback mechanism from D and R Sample a batch of transfer data and calculate the D3QN target value ; Calculate SA3C target value ; Step S4-4: Update the network parameters of the D3QN algorithm using the gradient descent optimization algorithm: Step S4-5: Update the critic network parameters of the SA3C algorithm by minimizing the loss L: Step S4-6: Use the sampled policy gradient to update the Actor network parameters of the SA3C algorithm: Step S4-7: Update transfer and Priority weights in D and R 、 , and replace the target neural network parameters of the D3QN algorithm at regular intervals And the SA3C target neural network parameters 、 .

Images