Personalized automatic lane changing and speed control driving collaborative decision-making method

Abstract

The invention relates to a personalized automatic lane changing and speed control driving collaborative decision-making method. The method comprises the following steps: S1, constructing a personalized decision-making hierarchical framework; s2, building a lane change decision training network, determining an optimization target of a lane change decision, introducing an action shielding mechanism to eliminate potential safety hazards of dangerous actions, setting an upper-layer state space, an action space and a lane change reward function required by training, and endowing the lane change reward function with personalized weights; s3, building a car-following decision training network, determining an optimization target of a car-following decision, quantizing the optimization target into a reward coefficient, setting a lower-layer state space, an action space and a car-following reward function required by training, and endowing the car-following reward function with personalized weights; and S4, a collaborative decision method is adopted, coupling training of the lane changing decision training network and the car following decision training network is carried out, and style reward priority experience playback is established to enrich experience playback priority measurement. Compared with the prior art, the method has the advantages of improving the safety and the driving efficiency of the vehicle and the like.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving, and in particular to a personalized automatic lane changing and speed control driving collaborative decision-making method. Background Art

[0002] With the rapid development of artificial intelligence, learning-based methods have been widely used in intelligent driving decision-making control systems, making driving behavior decisions have better generalization performance. Among them, deep reinforcement learning provides an intelligent solution for vehicle behavior decision-making by combining the representation advantages of deep learning with the reasoning advantages of reinforcement learning.

[0003] Autonomous driving decision control includes lane change decision and car following control. Lane change decision examines the driving relationship between vehicles on multiple lanes, while car following control focuses on the driving conditions of vehicles on the same lane. Traditionally, in autonomous driving decision control systems, lane change decision and car following control are studied as separate tasks, ignoring the potential cooperation between the two.

[0004] In the field of autonomous driving, there is a strong connection between lane changing and following a vehicle during driving, and the lane changing behavior will change the vehicle's trajectory and speed control strategy, so that the complete lane changing process involves the vehicle's lateral and longitudinal movement. In addition, factors such as the following distance and speed can directly affect the lane changing decision. If the following situation is poor, the vehicle may change lanes to seek a smoother route. Therefore, studying the synergistic relationship between lane changing and following a vehicle will help find the optimal action in a complex traffic environment and complete more accurate, efficient, and safe driving behavior decision-making control, thereby better realizing the driving intention and improving the safety and traffic efficiency of vehicle driving.

[0005] In recent years, in the field of deep reinforcement learning, although the intrinsic connection between lane changing and following has been considered, the differences in driving habits among drivers with different driving styles have been ignored, which may lead to large decision-making deviations. With the development of intelligent connected vehicles and advanced driver assistance systems (ADAS), driving style, as a comprehensive evaluation indicator of driving, is of great significance to the intelligence and safety of human-computer interaction capabilities. Different driving styles will affect the driver's decision-making behavior, thereby affecting the performance of autonomous driving vehicles. In autonomous driving decision control, considering the influence of driving style can make the decision-making method more in line with the driver's driving habits and improve driving safety and comfort.

[0006] In order to solve the technical problems existing in the above-mentioned fields, researchers are in urgent need of studying the synergistic relationship between the lane changing behavior and vehicle following control of driving decision control, optimizing and adjusting based on different driving styles, making more reasonable control decisions, and proposing a personalized automatic lane changing and speed control collaborative decision-making method. Summary of the invention

[0007] The purpose of the present invention is to provide a personalized automatic lane changing and speed control driving collaborative decision-making method.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A personalized automatic lane change and speed control driving collaborative decision-making method, comprising:

[0010] Step S1: constructing a personalized decision-making hierarchical framework, wherein the personalized decision-making hierarchical framework obtains characteristic parameters based on decision result simulation to provide a shared state space;

[0011] Step S2: building a lane-changing decision training network, determining the optimization target of the lane-changing decision, introducing an action shielding mechanism to eliminate the potential safety hazards of dangerous actions, setting the upper state space, upper action space and lane-changing reward function required for training, and assigning a personalized weight to the lane-changing reward function, wherein the upper state space is determined based on the shared state space;

[0012] Step S3: constructing a car-following decision training network, determining the optimization target of the car-following decision, and quantifying the optimization target into a reward coefficient, and setting the lower state space, lower action space and car-following reward function required for training, and giving a personalized weight to the car-following reward function, wherein the lower state space is determined based on the shared state space;

[0013] Step S4: adopt a collaborative decision-making method and perform coupled training of the lane change decision training network and the car-following decision training network, and build a style reward priority experience playback to enrich the experience playback priority metric.

[0014] The characteristic parameters include vehicle heading angle, headway, collision time, acceleration impact degree and lane change duration.

[0015] In the lane-changing decision training network, the optimization goal of the lane-changing decision is:

[0016] Traget lc =max(v e )+min(tlc)+min(dlc)

[0017] Among them: Traget lc is the optimization objective of lane-changing decision, v e is the vehicle's driving speed, tlc is the total number of lane changes, and dlc is the number of dangerous lane changes.

[0018] The action shielding mechanism includes:

[0019] After changing lanes to the target lane, the deceleration of the vehicle and the following vehicle cannot be less than the minimum safe deceleration to ensure that the vehicle has sufficient deceleration to avoid collision;

[0020] The distance between vehicles ensures a safe distance:

[0021]

[0022] Where: d max is the maximum distance between the vehicle and the vehicle in front, d safe is the safe following distance at the current speed of the vehicle, t b is the driver's reaction time, a safe It is the minimum safe deceleration on the road.

[0023] The upper state space is:

[0024] LC_State = {v e ,a e ,{v pi ,a pi ,h i ,pd i , pv i}}

[0025] Where: LC_State is the upper state space, a e is the acceleration of the vehicle, v pi is the speed of the front vehicle in lane i, a pi is the acceleration of the front vehicle in lane i, h i is the headway time between the vehicles in lane i, pd i is the position advantage with the vehicle in lane i, pv i is the speed advantage of each vehicle in lane i;

[0026] The upper action space is:

[0027] LC_Action = {0,1,2}

[0028] Among them: LC_Action is the upper-level action space, 0 means keeping the current lane, 1 means changing lanes to the left, and 2 means changing lanes to the right;

[0029] The lane-changing reward function is:

[0030] LC_Reward=α*v e +γ*lc+ω*c

[0031] Among them: LC_Reward is the lane change reward function, lc is whether the lane change behavior occurs, c is the number of collisions, α, γ, ω are weights.

[0032] The position advantage with the vehicle in lane i is:

[0033]

[0034] Where: y e is the longitudinal distance between the vehicle and the vehicle behind in lane i when the vehicle decides to change lanes, y ri is the longitudinal distance between the vehicle and the vehicle behind in lane i when the vehicle decides not to change lanes, v ri is the speed of the car behind in lane i, t b for driver reaction time;

[0035] The speed advantage of each vehicle in lane i is:

[0036]

[0037] Among them: LC prob is the feasibility of the vehicle’s lane change behavior, v pi is the speed of the front vehicle in lane i, t 1i The time required to accelerate to the maximum speed limit of the road, t 2i The time to maintain the maximum speed limit, t 3i The time required to decelerate to the speed of the vehicle ahead.

[0038] The optimization goal of the car-following decision training network and the car-following decision is:

[0039]

[0040] Among them: Traget cf is the optimization goal of the car-following decision, v e is the vehicle speed, a c is the acceleration of the front vehicle, and c is the number of collisions that occur during the following process.

[0041] The lower state space is:

[0042] F_State = {v e ,v p ,a p ,h}

[0043] Where: F_State is, v p is the speed of the vehicle ahead in the current lane, a p is the acceleration of the vehicle in front of the front lane, h is the distance between the leading vehicle and the vehicle in front when following in the current lane;

[0044] The lower action space is:

[0045] CF_Action = {a e ∈[a dec ,a acc]}

[0046] Among them: CF_Action is the lower layer action space, a e is the acceleration of the vehicle, a dec is the maximum deceleration of the vehicle, a acc is the maximum acceleration of the vehicle;

[0047] The following reward function is:

[0048]

[0049] Among them: CF_Reward is the following reward function, β, λ, μ are weights, is the mean square of the vehicle acceleration.

[0050] The style reward priority experience playback divides the experience priority calculation into TD error index and incentive index, where the incentive index consists of R_Advantage and Q_Advantage, which are used to evaluate different driving styles for different rewards and speed exploration preferences and improve the convergence of the algorithm. Based on the reward expectations and target preferences under different styles, the sample priority is given a personalized weight coefficient. For different transfers (s t ,a t ,r t ,s t+1 ),, its sample priority P(t) is defined as:

[0051] P(t)=ρ*P TD (t)+(1-ρ)*P I (t)+ε

[0052]

[0053] Where: ρ is the weight of the TD error indicator, P TD (t) is the TD error index, P I (t) is the incentive index, ε is a trace constant, is the average TD error, η 1 , η 2 is a constant, rt is a short-term reward, Q(s t ,a t ) is a long-term reward;

[0054] The average TD error is:

[0055]

[0056] Where: Q traget (s t ,a t ) is the TD target value calculated based on the reward obtained from the environment, Q(st ,a t θ j ) is the Action-value function value Q estimated by the Critic network j in the car-following decision training network.

[0057] The car-following decision training network is built based on the SA3C algorithm, and the lane-changing decision training network is built based on the D3QN algorithm;

[0058] The coupling training includes:

[0059] Step S4-1: Initialize the neural network parameters for the D3QN and SA3C algorithms, including:

[0060] Random initial network parameters: θ and θ′←θ; set the maximum capacity U of the replay buffer d and the number of training batches B d ,

[0061] Initialize the parameters of SA3C, including the Actor network parameters θ μ 、Critic network parameter φ i (i=1,2,3) and target network parameters θ μ′ ←θ μ ,φ′ i ←φ i ; Set the maximum capacity of the playback buffer U r and the number of training batches B r ;

[0062] Step S4-2: In each training round, select and execute actions according to the current strategy, and accept the initial observation state s from the simulation environment:

[0063] (1) Process the Q value of the dangerous action according to the current state, select and execute the action

[0064] If the action corresponding to the maximum value of the current Q(s,a|θ) is the safest, then the current action is selected. Otherwise, with a certain probability Randomly select an action

[0065] (2) Use exploration noise ζ to select actions

[0066] Execute an action and By calculating the reward function and observing the reward r t and the next state s t+1 ,

[0067] (3) Transfer and The data are stored in the playback lane change decision buffer D and the following decision buffer R respectively;

[0068] Step S4-3, use SIPER from and Sample a batch of transfer data and calculate the D3QN target value y j =r j +γQ(s′,argmaxQ(s′,a′|θ);θ′); calculate the SA3C target value y j =r j +γQ(mst i＝1,2,3 Q(s′,a′|φ i )-αlogμ(s′|θ μ )).

[0069] Step S4-4: Update the network parameters of the D3QN algorithm through the gradient descent optimization algorithm:

[0070]

[0071] Step S4-5: Update the Criti of the SA3C algorithm by minimizing the loss L c Network parameters:

[0072]

[0073] Step S4-6: Use sampling policy gradient to update the Actor network parameters of the SA3C algorithm:

[0074]

[0075] Step S4-7: Update transfer and Priority weights in D and R And replace the target neural network parameter θ of the D3QN algorithm at regular intervals ′ and the SA3C target neural network parameters θ μ ,φ i ′ .

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] 1. The present invention proposes a personalized decision-making plan with customized reward function weights to adapt to the needs and preferences of different drivers.

[0078] 2. The present invention adopts hierarchical reinforcement learning to achieve collaborative decision-making, which consists of a D3QN planner with dangerous lane-changing behavior and an improved SA3C car-following planner with three evaluation networks to improve stability.

[0079] 3. The present invention takes into account the reward preference and strategic vision under different driving styles, and proposes a style-incentive priority experience replay mechanism (SIPER), which integrates immediate rewards and long-term returns into the experience priority calculation and assigns different sample priority weight coefficients.

[0080] 4. The present invention divides lane advantage into speed advantage and position advantage, and incorporates them into the shared state space to reflect the interaction between the intelligent agent and the surrounding vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is a schematic diagram of the hierarchical decision-making framework process;

[0082] Figure 2 Schematic diagram of the decision-making process in a complex traffic environment;

[0083] Figure 3 This is a schematic diagram of the hierarchical collaborative decision-making framework structure;

[0084] Figure 4 It is a schematic diagram of the main steps of the present invention. DETAILED DESCRIPTION

[0085] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0086] On the whole, the present invention takes automatic lane changing and speed control of different driving styles as the research object, and develops a personalized driving decision framework based on the driving style perception decision method of deep reinforcement learning. In addition, a personalized hierarchical decision framework for lane changing and following is built based on the D3QN algorithm and the flexible actor-critic (SA3C) algorithm, and the coupling training of the two is carried out to achieve the collaborative control effect of following and lane changing. The main steps include extracting driving behavior features instead of calculating statistical features; building a traffic simulation environment, including lane change decisions, following vehicle control and multi-vehicle interaction; building a personalized hierarchical decision framework for lane changing and following, building an actor (Actor) and critic (Critic) network and a target critic network to form an intelligent agent; training the hierarchical driving strategy, and obtaining an inheritable network strategy through the interaction between the environment and the intelligent agent; processing the trajectory driving data, conducting trajectory experiments, and simulating training.

[0087] This method uses D3QN with priority experience replay as the core of lane-changing decision-making, combines the advantage function with the lane-changing benefit function, and introduces an action shielding mechanism to gradually eliminate the safety hazards of dangerous actions; SA3C is used to optimize the maximum expected reward and policy entropy, encourage exploration and improve training stability. In addition, by analyzing the behavioral characteristics of the driver, considering the impact of different driving styles, and meeting personalized driving needs, it can help the vehicle make more reasonable lane-changing decisions and follow-up control.

[0088] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the application is further elaborated in detail below in conjunction with the accompanying drawings. The described examples are only part of the embodiments involved in this patent. All non-innovative examples of other researchers in this field on this embodiment are within the scope of protection of this patent.

[0089] A personalized automatic lane change and speed control driving collaborative decision-making method, such as Figure 4 and Figure 3 As shown, including:

[0090] Step S1: Construct a personalized decision-making hierarchical framework, where Figure 1 As shown, the personalized decision hierarchical framework obtains characteristic parameters based on decision result simulation to provide a shared state space;

[0091] Specifically, according to the differences in driving decisions of different driving styles, a personalized hierarchical decision-making framework is built, in which the simulation environment is a lane change and following decision environment in a multi-lane scenario under mixed traffic flow:

[0092] The driving style characteristics of lateral and longitudinal vehicle motion are extracted from the autonomous driving style, and the influence of human factors is taken into account. According to the existing natural driving data research, the vehicle heading angle, headway, collision time, acceleration impact degree and lane change duration are mainly selected as characteristic parameters.

[0093] In the decision-making goal design, the lane-changing scenario process is as follows: Figure 2 As shown in the figure. In free lane change, when the vehicle is blocked by the vehicle in front and cannot obtain the expected speed gain in the current lane, the vehicle will consider changing lanes. At this time, the vehicle will weigh the driving conditions and safety advantages and disadvantages of the adjacent lanes. At the same time, in order to prevent vehicles from changing lanes frequently and arbitrarily, the number of lane changes is controlled within a reasonable range through a penalty mechanism.

[0094] Step S2: Build a lane-changing decision training network, determine the optimization goal of the lane-changing decision, introduce an action shielding mechanism to eliminate the safety hazards of dangerous actions, set the upper state space, upper action space and lane-changing reward function required for training, and give the lane-changing reward function a personalized weight, wherein the upper state space is determined based on the shared state space;

[0095] Lane changing process Figure 2 As shown. In free lane change, when the vehicle is blocked by the vehicle in front and cannot obtain the expected speed gain in the current lane, the vehicle will consider changing lanes. At this time, the vehicle will weigh the driving conditions and safety advantages and disadvantages of the adjacent lanes. At the same time, in order to prevent vehicles from changing lanes frequently and arbitrarily, the number of lane changes is controlled within a reasonable range through a penalty mechanism. Therefore, in the lane change decision training network, the optimization goal of the lane change decision is:

[0096] Traget lc =max(v e )+min(tlc)+min(dlc)

[0097] Among them: Traget lc is the optimization objective of lane-changing decision, v e is the vehicle's driving speed, tlc is the total number of lane changes, and dlc is the number of dangerous lane changes.

[0098] Action shielding mechanisms include:

[0099] After changing lanes to the target lane, the deceleration of the vehicle and the following vehicle cannot be less than the minimum safe deceleration to ensure that the vehicle has sufficient deceleration to avoid collision. Specifically:

[0100]

[0101] Among them: LC prob is the feasibility of vehicle lane change, a′ e is the deceleration of the vehicle after changing lanes, a safe is the minimum safe deceleration on the road, a′ en is the deceleration of the vehicle behind the target lane,

[0102] At the same time, in order to pursue higher driving efficiency, the distance between vehicles should not be too large, and a safe distance should be ensured. The distance between vehicles ensures a safe distance:

[0103]

[0104] Where: d max is the maximum distance between the vehicle and the vehicle in front, d safe is the safe following distance at the current speed of the vehicle, t b The driver's reaction time.

[0105] The upper state space is:

[0106] LC_State = {v e ,a e ,{v pi , a pi ,h i ,pdi ,pv i}}

[0107] Where: LC_State is the upper state space, a e is the acceleration of the vehicle, v pi is the speed of the front vehicle in lane i, a pi is the acceleration of the front vehicle in lane i, h i is the headway time between the vehicles in lane i, pd i is the position advantage with the vehicle in lane i, pv i is the speed advantage of each vehicle in lane i;

[0108] Among them, in the lane-changing scenario, the main considerations are the distance between the front and rear vehicles in the target lane, and the competitive relationship between the rear vehicle and the vehicle in the target lane.

[0109] When the vehicle decides to change lanes, the aggressive rear vehicle will speed up to protect its rights and interests, narrow the gap with the front vehicle, and thus prevent the main vehicle from changing lanes. The conservative rear vehicle will slow down to avoid collision and widen the gap, thus giving way to the vehicle in the lane to change lanes. Therefore, position advantage is introduced to represent the competitive relationship between vehicles. Specifically, the position advantage with the vehicle in the i-th lane is:

[0110]

[0111] Where: y e is the longitudinal distance between the vehicle and the vehicle behind in lane i when the vehicle decides to change lanes, y ri is the longitudinal distance between the vehicle and the vehicle behind in lane i when the vehicle decides not to change lanes, v ri is the speed of the car behind in lane i, t b The driver's reaction time is also used as the headway safety threshold, which is generally 1.5 seconds.

[0112] In the following car scenario, the main consideration is the vehicle in front of the current lane. If the vehicle in front is aggressive and driving faster, the maximum speed advantage that the vehicle can obtain will also be expanded. If the vehicle in front is conservative and driving slower, the vehicle cannot drive too fast, thus obtaining a lower speed advantage. The speed advantage is calculated under the assumption that the vehicle can catch up with the vehicle in front of it in the lane as quickly as possible. Specifically, the speed advantage with each vehicle in the i-th lane is:

[0113]

[0114] Where: v pi is the speed of the front vehicle in lane i, t 1i The time required to accelerate to the maximum speed limit of the road, t 2i The time to maintain the maximum speed limit, t 3i The time required to decelerate to the speed of the vehicle ahead.

[0115]

[0116] By 1i and t 3i After the stage, the distance between the vehicle and the vehicle in front is:

[0117] s di =(t 1i +t 3i )*v pi +h i -s 1i -s 3i

[0118] When di When it is less than 0, the maximum speed that the ego vehicle can reach is less than the maximum speed limit of the road, and there is no stage of maintaining the maximum speed limit. At this time, the acceleration and deceleration process of the ego vehicle is calculated as follows:

[0119]

[0120] Among them, v′ max is the maximum allowed speed on the lane, t 1 ′ i is the time it takes for the vehicle to accelerate to the maximum allowed speed, t 3 ′ i It is the time to decelerate from the maximum allowed speed to the speed of the vehicle ahead.

[0121] The upper action space is:

[0122] LC_Action = {0,1,2}

[0123] Among them: LC_Action is the upper-level action space, 0 means keeping the current lane, 1 means changing lanes to the left, and 2 means changing lanes to the right;

[0124] The lane-changing reward function is:

[0125] LC_Reward=α*v e +γ*lc+ω*c

[0126] Among them: LC_Reward is the lane change reward function, lc is whether the lane change behavior occurs, c is the number of collisions, α, γ, ω are weights.

[0127] Step S3: Building a car-following decision-making training network, determining the optimization target of the car-following decision, and quantifying the optimization target into a reward coefficient, and setting the lower state space, lower action space and car-following reward function required for training, and giving a personalized weight to the car-following reward function, wherein the lower state space is determined based on the shared state space;

[0128] The optimization goal of the car-following decision training network is:

[0129]

[0130] Among them: Traget cf is the optimization goal of the car-following decision, v e is the vehicle speed, a c is the acceleration of the front vehicle, and c is the number of collisions that occur during the following process.

[0131] The lower state space is:

[0132] F_State = {v e ,v p ,a p ,h}

[0133] Where: F_State is, v p is the speed of the vehicle ahead in the current lane, a p is the acceleration of the vehicle in front of the front lane, h is the distance between the leading vehicle and the vehicle in front when following in the current lane;

[0134] The lower action space is:

[0135] CF_Action = {a e ∈[a dec ,a acc ]}

[0136] Among them: CF_Action is the lower layer action space, a e is the acceleration of the vehicle, a dec is the maximum deceleration of the vehicle, a acc is the maximum acceleration of the vehicle;

[0137] The following reward function is:

[0138]

[0139] Among them: CF_Reward is the following reward function, β, λ, μ are weights, is the mean square of the vehicle acceleration.

[0140] Personalized driving hierarchical coordination decision-making process Figure 3As shown. First, the agent obtains the current driving state information of itself and surrounding vehicles, as well as the advantage variables of adjacent lanes, which can represent the vehicle's own driving state and the interaction between the agent and surrounding vehicles as a shared state space entering the decision module. The decision framework consists of upper and lower planners. The upper planner then makes personalized lane changing decisions based on the required state information and the competitive advantage relative to the surrounding vehicles. When the agent decides to maintain the current lane, the state information of the current lane is transmitted to the following planner. The lower planner will provide a continuous speed control strategy that matches the driving style for the next step. Otherwise, it adopts a lane changing behavior and updates the state space of the following planner using the state variables of the target lane.

[0141] Step S4: adopt a collaborative decision-making method and perform coupled training of the lane change decision training network and the car-following decision training network, and build a style reward priority experience playback to enrich the experience playback priority metric.

[0142] Style Reward Priority Experience Replay divides the experience priority calculation into TD error index and incentive index, where the incentive index consists of R_Advantage and Q_Advantage, which are used to evaluate different driving styles for different rewards and speed exploration preferences and improve the convergence of the algorithm. Based on the reward expectations and target preferences under different styles, the sample priority is given a personalized weight coefficient. For different transfers (s t ,a t ,r t ,s t+1 ),, its sample priority P(t) is defined as:

[0143] P(t)=ρ*P TD (t)+(1-ρ)*P I (t)+ε

[0144]

[0145]

[0146] Where: ρ is the weight of the TD error indicator, P TD (t) is the TD error index, P I (t) is the incentive index, ε is a trace constant, is the average TD error, η 1 , η 2 is a constant, rt is a short-term reward, Q(s t ,a t ) is a long-term reward;

[0147] The average TD error is:

[0148]

[0149] Where: Q traget (s t ,a t ) is the TD target value calculated based on the reward obtained from the environment, Q(s t ,a t θ j ) is the Action-value function value Q estimated by the Critic network j in the car-following decision training network.

[0150] Assume that the experience in the buffer follows the probability distribution p ψ ~φ, then the priority probability of transfer (s t ,a t ,r t ,s t+1 ) is calculated as follows:

[0151]

[0152] While inheriting the traditional PER and tree experience storage structure, efficient performance level management improves the convergence speed, but also changes the sampling experience distribution and introduces potential estimation bias. The error is corrected by assigning sample importance sampling weights (ISW):

[0153]

[0154] Among them, ω t represents the transfer ISW with priority probability p(t), bs represents the batch size, and β represents the importance sampling correction index, which increases from the initial value β over time. 0 Linearly degenerates to 1. ω t Can be done by using Applied in Q-learning updates.

[0155] In addition, the car-following decision training network is built based on the SA3C algorithm, and the lane-changing decision training network is built based on the D3QN algorithm;

[0156] Coupling training includes:

[0157] Step S4-1: Initialize the neural network parameters for the D3QN and SA3C algorithms, including:

[0158] Random initial network parameters: θ and θ ′ ←θ; Set the maximum capacity U of the replay buffer d and the number of training batches B d ,

[0159] Initialize the parameters of SA3C, including the Actor network parameters θ μ 、Critic network parameter φ i(i=1,2,3) and target network parameters Set the maximum capacity of the playback buffer U r and the number of training batches B r ;

[0160] Step S4-2: In each training round, select and execute actions according to the current strategy, and accept the initial observation state s from the simulation environment:

[0161] (1) Process the Q value of the dangerous action according to the current state, select and execute the action

[0162] If the action corresponding to the maximum value of the current Q(s,a|θ) is the safest, then the current action is selected. Otherwise, with a certain probability Randomly select an action

[0163] (2) Use exploration noise ζ to select actions

[0164] Execute an action and By calculating the reward function and observing the reward r t and the next state s t+1 ,

[0165] (3) Transfer and The data are stored in the playback lane change decision buffer D and the following decision buffer R respectively;

[0166] Step S4-3, use SIPER from and Sample a batch of transfer data and calculate the D3QN target value y j =r j +γQ(s′,argmaxQ(s′,a′|θ);θ′); calculate the SA3C target value y j =r j +γQ(mst i＝1,2,3 Q(s′,a′|φ i )-αlogμ(s′|θ μ )).

[0167] Step S4-4: Update the network parameters of the D3QN algorithm through the gradient descent optimization algorithm:

[0168]

[0169] Step S4-5, update the critic network parameters of the SA3C algorithm by minimizing the loss L:

[0170]

[0171] Step S4-6: Use sampling policy gradient to update the Actor network parameters of the SA3C algorithm:

[0172]

[0173] Step S4-7: Update transfer and Priority weights in D and R And replace the target neural network parameter θ of the D3QN algorithm at regular intervals ′ and the SA3C target neural network parameters θ μ ,φ i ′ .

[0174] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

Claims

1. A personalized automatic lane change and speed control driving collaborative decision-making method, characterized in that: include: Step S1: constructing a personalized decision-making hierarchical framework, wherein the personalized decision-making hierarchical framework obtains characteristic parameters based on decision result simulation to provide a shared state space; Step S2: building a lane-changing decision training network, determining the optimization target of the lane-changing decision, introducing an action shielding mechanism to eliminate the potential safety hazards of dangerous actions, setting the upper state space, upper action space and lane-changing reward function required for training, and assigning a personalized weight to the lane-changing reward function, wherein the upper state space is determined based on the shared state space; Step S3: constructing a car-following decision training network, determining the optimization target of the car-following decision, and quantifying the optimization target into a reward coefficient, and setting the lower state space, lower action space and car-following reward function required for training, and giving a personalized weight to the car-following reward function, wherein the lower state space is determined based on the shared state space; Step S4: adopt a collaborative decision-making method and perform coupled training of the lane change decision training network and the car-following decision training network, and build a style reward priority experience playback to enrich the experience playback priority metric.

2. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 1, characterized in that: The characteristic parameters include vehicle heading angle, headway, collision time, acceleration impact degree and lane change duration.

3. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 1, characterized in that: In the lane-changing decision training network, the optimization goal of the lane-changing decision is: raft lc =max(v e )+min(lc)+min(dlc) Among them: Traget lc is the optimization objective of lane-changing decision, v e is the vehicle's driving speed, tlc is the total number of lane changes, and dlc is the number of dangerous lane changes.

4. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 3, characterized in that: The action shielding mechanism includes: After changing lanes to the target lane, the deceleration of the vehicle and the following vehicle cannot be less than the minimum safe deceleration to ensure that the vehicle has sufficient deceleration to avoid collision; The distance between vehicles ensures a safe distance: Where: d max is the maximum distance between the vehicle and the vehicle in front, d safe is the safe following distance at the current speed of the vehicle, t b is the driver's reaction time, a safe It is the minimum safe deceleration on the road.

5. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 3, characterized in that: The upper state space is: LC_State={v e ,a e ,{v pi ,a pi ,h i ,pd i ,pv i }} Where: LC_State is the upper state space, a e is the acceleration of the vehicle, v pi is the speed of the front vehicle in lane i, a pi is the acceleration of the front vehicle in lane i, h i is the headway time between the vehicles in lane i, pd i is the position advantage with the vehicle in lane i, pv i is the speed advantage of each vehicle in lane i; The upper action space is: LC_Action = {0,1,2} Among them: LC_Action is the upper-level action space, 0 means keeping the current lane, 1 means changing lanes to the left, and 2 means changing lanes to the right; The lane-changing reward function is: LC_Reward=α*v e +γ*lc+ω*c Among them: LC_Reward is the lane change reward function, lc is whether the lane change behavior occurs, c is the number of collisions, α, γ, ω are weights.

6. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 5, characterized in that: The position advantage with respect to the vehicle in lane i is: Where: y e is the longitudinal distance between the vehicle and the vehicle behind in lane i when the vehicle decides to change lanes, y ri is the longitudinal distance between the vehicle and the vehicle behind in lane i when the vehicle decides not to change lanes, v ri is the speed of the car behind in lane i, t b for driver reaction time; The speed advantage of each vehicle in lane i is: Among them: LC prob is the feasibility of the vehicle’s lane change behavior, v pi is the speed of the front vehicle in lane i, t 1i The time required to accelerate to the maximum speed limit of the road, t 2i The time to maintain the maximum speed limit, t 3i The time required to decelerate to the speed of the vehicle ahead.

7. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 1, characterized in that: The optimization goal of the car-following decision training network and the car-following decision is: Among them: Traget cf is the optimization goal of the car-following decision, v e is the vehicle speed, a c is the acceleration of the front vehicle, and c is the number of collisions that occur during the following process.

8. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 7, characterized in that: The lower state space is: F_Stare={v e ,v p ,a p ,h} Among them: F_Stare is, v p is the speed of the vehicle ahead in the current lane, a p is the acceleration of the vehicle in front of the front lane, h is the distance between the leading vehicle and the vehicle in front when following in the current lane; The lower action space is: CF_Action={a e ∈[a dec ,a acc ]} Among them: CF_Action is the lower layer action space, a e is the acceleration of the vehicle, a dec is the maximum deceleration of the vehicle, a acc is the maximum acceleration of the vehicle; The following reward function is: CF_Reward=β*v e +λ*J e +μ*d Among them: CF_Reward is the following reward function, β, λ, μ are weights, v e is the speed of the vehicle, J e is the vehicle acceleration impact degree, and d is the distance between the vehicle and the front vehicle.

9. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 7, characterized in that: The style reward priority experience playback divides the experience priority calculation into TD error index and incentive index, where the incentive index consists of R_Advantage and Q_Advantage, which are used to evaluate different driving styles for different rewards and speed exploration preferences and improve the convergence of the algorithm. Based on the reward expectations and target preferences under different styles, the sample priority is given a personalized weight coefficient. For different transfers (s t ,a t ,r t ,s t+1 ),, its sample priority P(t) is defined as: P(t)=ρ*P TD (t)+(1-ρ)*P I (t)+e Where: ρ is the weight of the TD error indicator, P TD (t) is the TD error index, P I (t) is the incentive index, ε is a trace constant, is the average TD error, η1 and η2 are constants, rt is the short-term reward, Q(s t ,a t ) is a long-term reward; The average TD error is: Where: Q traget (s t ,a t ) is the TD target value calculated based on the reward obtained from the environment, Q(s t ,a t θ j ) is the Action-value function value Q estimated by the Critic network j in the car-following decision training network.

10. The personalized automatic lane change and speed control driving collaborative decision-making method according to claim 9, characterized in that: The car-following decision training network is built based on the SA3C algorithm, and the lane-changing decision training network is built based on the D3QN algorithm; The coupling training includes: Step S4-1: Initialize the neural network parameters for the D3QN and SA3C algorithms, including: Random initial network parameters: θ and θ′←θ; set the maximum capacity U of the replay buffer d and the number of training batches B d , Initialize the parameters of SA3C, including the Actor network parameters θ μ 、Critic network parameter φ i (i=1,2,3) and target network parameters θ μ′ ←θ μ ,φ i ′←φ i ; Set the maximum capacity of the playback buffer U r and the number of training batches B r ; Step S4-2: In each training round, an action is selected and executed according to the current strategy, and an initial observation state s is received from the simulation environment, including: Process the Q value of the dangerous action according to the current state, select and execute the action If the action corresponding to the maximum value of the current Q(s,a|θ) is the safest, then the current action is selected. Otherwise, with a certain probability Randomly select an action Use Exploration Noise Select the action to perform Execute an action and By calculating the reward function and observing the reward r t and the next state s t+1 , Will transfer and The data are stored in the playback lane change decision buffer D and the following decision buffer R respectively; Step S4-3, use SIPER from and Sample a batch of transfer data and calculate the D3QN target value y j =r j +γQ(s′,argmaxQ(s′,a′|θ);θ′); calculate the SA3C target value y j =r j +γQ(mst i＝1,2,3 Q(s′,a′|φ i )-αlogμ(s′|θ μ )). Step S4-4: Update the network parameters of the D3QN algorithm through the gradient descent optimization algorithm: Step S4-5, update the critic network parameters of the SA3C algorithm by minimizing the loss L: Step S4-6: Use sampling policy gradient to update the Actor network parameters of the SA3C algorithm: Step S4-7: Update transfer and Priority weights in D and R And replace the target neural network parameters θ′ of the D3QN algorithm and the target neural network parameters θ of the SA3C algorithm at regular intervals μ ,φ′ i .

Images