A multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning

Abstract

The present application belongs to the field of intelligent driving technology, and specifically relates to a multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning, including step 1: obtaining the observation states of each environment in the driving simulation environment; step 2: establishing the overall environment state space based on the observation states of each environment, and establishing the driving right allocation coefficient action space based on the driving right allocation strategy; step 3: based on step 2, it is divided into the first layer of lane keeping model training objectives and the second layer of left and right lane changing model training objectives; step 4: establishing a basic reward function, integrating the driving of the fully automatic driving system, the full driver model and the human-machine co-driving to obtain the fusion reward function of the lane keeping model and the left and right lane changing model; step 5: hierarchical training of the fusion reward function until the optimal human-machine shared driving system driving right allocation strategy is obtained and the allocation weight is output. The present application simplifies the process of allocating driving control rights in the human-machine shared driving system, and improves the efficiency and accuracy of training.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and specifically relates to a multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning. Background Art

[0002] The human-machine co-driving technology of smart cars is an effective transitional means to solve the problem that its intelligence level is difficult to quickly leap to a highly automated level. Compared with the manual driving mode, the human-machine co-driving mode can reduce the accident rate and solve the ethical issues brought about by autonomous driving to a certain extent. The shared human-machine co-driving structure means that in most cases, the driver and the autonomous driving system have the authority to perceive, make decisions and control at the same time, and the operating instructions to which the vehicle responds come from both parties. Therefore, how to dynamically allocate driving control rights between the driver and the autonomous driving system in L3 shared autonomous driving vehicles becomes the key.

[0003] Patent CN116142292A proposes a control method suitable for a human-machine co-driving steering system. The system consists of an internal torque assist mode, an external angle execution mode, and an external angle compensation mode. Internal torque assist mode: The driver directly and completely controls the steering action of the steering wheel, and the human-machine co-driving steering system gives the driver different steering feel and road feel feedback; External angle execution mode: The human-machine co-driving steering system independently executes the steering angle signal from ADAS or ADS to perform lateral control of the vehicle; External angle compensation mode: Compensate for the steering angle input by the driver, and dynamically allocate the driving weights of the human-machine co-driving steering system and the driver. Although this method solves the dynamic allocation of lateral shared driving rights, it does not consider the dynamic allocation of longitudinal shared driving rights during driving.

[0004] Patent CN115489585A proposes a human-machine co-driving wire steering control system suitable for smart cars, aiming to solve the problem of selecting the steering wheel control mode in different manual driving modes, automatic driving modes, and human-machine shared driving modes. A total of five steering wheel control modes are designed, and the fuzzy control method is combined to select the appropriate steering wheel control mode. This method also only solves the dynamic allocation of lateral shared driving rights, and the fuzzy control method used is easily affected by the prior knowledge modeling error, resulting in low algorithm adaptability.

[0005] Therefore, in the L3 shared human-machine co-driving process, how to smoothly distribute the lateral and longitudinal driving control rights between the driver and the autonomous driving system, while being able to adapt to the dynamic changes of the environment is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of the defects of the prior art, the present invention proposes a multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning, which uses the hierarchical deep reinforcement learning method to solve the problem of continuous dynamic allocation of lateral and longitudinal driving control rights between the driver and the automatic driving system in the human-machine shared driving system of an intelligent vehicle, simplifies the driving control rights allocation process of the human-machine shared driving system, establishes a lane keeping model and a left-right lane changing model by setting multiple objectives, and performs hierarchical training on the lane keeping model and the left-right lane changing model, thereby improving the training efficiency and accuracy.

[0007] In order to achieve the technical purpose of the present invention, the present invention will adopt the following technical solutions:

[0008] A multi-objective vehicle-to-people sharing control method based on hierarchical deep reinforcement learning, the multi-objective vehicle-to-people sharing control method based on hierarchical deep reinforcement learning comprises the following steps:

[0009] Step 1: The human-machine shared driving system obtains various environmental observation states of the autonomous driving vehicle in the driving simulation environment through sensor equipment. Each environmental observation state at least includes the vehicle's own state, the vehicle's surrounding environment state, and the driver's control input state;

[0010] Step 2: Based on the environmental observation states in step 1, establish the overall environmental state space and the vehicle kinematic model that describes the lateral and longitudinal dynamic behaviors of the vehicle, and based on the driving rights allocation strategy of the human-vehicle shared driving system, establish the driving rights allocation coefficient action space of the driver and the automatic driving system;

[0011] Step 3: Use hierarchical reinforcement learning to divide the overall environmental state space and driving right allocation coefficient action space of the human-vehicle shared driving system obtained in step 2 into two layers of training targets. The first layer of training targets is the lane keeping model, and the second layer of training targets is the left and right lane changing model.

[0012] Step 4: Based on the vehicle kinematic model in step 2 and the lane keeping model and left and right lane changing models decomposed in step 3, basic reward functions are established respectively, and they are integrated according to the three modes of fully automated driving system driving, full driver model and human-machine co-driving to obtain the lane keeping model fusion reward function and the left and right lane changing model fusion reward function;

[0013] Step 5: Based on the lane keeping model fusion reward function and the left and right lane changing model fusion reward function established in step 4, training is performed. At the same time, a driving simulation environment required for the lane keeping model and the left and right lane changing model training is established. The overall environmental state space and driving right allocation coefficient action space obtained by the three modes of fully automatic driving system driving, fully driver model driving and human-machine co-driving are simultaneously input into the driving simulation environment and fusion reward function of the lane keeping model for training. The trained model is input into the driving simulation environment and fusion reward function of the left and right lane changing model for further training until the optimal driving right allocation strategy for the human-machine shared driving system is obtained through training, and the allocation weight is output.

[0014] Furthermore, in step 2, an overall environment state space S = [S1, S2, S3] is established based on each environment observation state in step 1;

[0015] Where: S1 is the state space of the vehicle itself; S2 is the state space of the vehicle's surrounding environment; S3 is the driver's control input state space;

[0016] In step 2, the driving rights allocation strategy of the human-vehicle shared driving system is used to establish the driving rights allocation coefficient action space between the driver and the automatic driving system.

[0017] Where: It represents the lateral driving right allocation coefficient of the automatic driving system; Indicates the driver's lateral driving right allocation coefficient; It represents the longitudinal driving right allocation coefficient of the automatic driving system; Indicates the driver's longitudinal driving right allocation coefficient; and

[0018] Furthermore, the vehicle's own state space Vehicle surroundings state space Driver control input state space

[0019] Where: Indicates the lateral position coordinates of the vehicle at the current moment; Indicates the longitudinal position coordinates of the vehicle at the current moment; Indicates the current speed of the vehicle; Indicates the acceleration input by the self-driving system at the current moment; Indicates the steering angle input by the self-driving system at the current moment; Indicates the comprehensive acceleration of the vehicle after the driving right is allocated at the current moment; Indicates the comprehensive steering angle of the current driving right distribution and the rear vehicle; φt Indicates the heading angle of the vehicle at the current moment; Indicates the relative lateral position of the vehicle and the vehicle in front of the lane at the current moment; Indicates the relative lateral position of the vehicle and the vehicle behind it in the adjacent lane at the current moment; Indicates the relative lateral position of the vehicle and the preceding vehicle in the adjacent lane at the current moment; Indicates the relative longitudinal position of the vehicle and the vehicle behind it in the adjacent lane at the current moment; Indicates the relative longitudinal position of the vehicle and the preceding vehicle in the adjacent lane at the current moment; Indicates the current speed of the vehicle ahead in the lane; Indicates the current driving speed of the vehicle ahead in the adjacent lane; L indicates lane line information; T indicates traffic signal information; Indicates the steering angle input by the driver through the steering wheel at the current moment; Indicates the acceleration input by the driver through the brake and accelerator at the current moment.

[0020] Furthermore, the lane keeping model in step 3 includes the environment state space S k and driving rights allocation coefficient action space A, The environment state space S of the lane keeping model k a t Include and The left-right lane-changing model in step 3 contains the environment state space S c and driving rights allocation coefficient action space A, The environmental state space S of the left-right lane-changing model c δ t Include and

[0021] In step 3, the driving right allocation coefficient action space A of the lane keeping model is the same as the driving right allocation coefficient action space A of the left-right lane changing model.

[0022] Furthermore, step 4 specifically includes:

[0023] Step 4.1: Establish the basic reward function R of the lane keeping model k , R k Determined by the following formula:

[0024]

[0025] Where: Represents the reward function for approaching the end point; represents the efficiency reward function; represents the lane keeping reward function; represents the reward function for following the rules;

[0026] Step 4.2: Establish the basic reward function R for the left and right lane change model c , R c Determined by the following formula:

[0027]

[0028] Where: represents the reward function for successful lane change; represents the efficiency reward function; represents the distance reward function; represents the comfort reward function;

[0029] Step 4.3: Based on the basic reward functions of steps 4.1 and 4.2, the lane keeping model fusion reward function is obtained by integrating the three driving modes: fully automated driving system driving, fully driver model driving, and human-machine co-driving. Fusion reward function with left and right lane change model

[0030] Furthermore, the reward function close to the end point in step 4.1 is By formula Sure;

[0031] The efficiency reward function is given by the formula OK, when When , the training is terminated;

[0032] The lane keeping reward function is given by the formula OK, when When , the training is terminated;

[0033] The rule-following reward function is given by the formula Sure, Terminate training;

[0034] In the formula, Indicates the current speed of the vehicle; according to three different driving modes: fully automatic driving system driving, fully driver model driving, and human-machine co-driving; a t Corresponding to Indicates the acceleration input by the self-driving system at the current moment; Indicates the steering angle input by the self-driving system at the current moment; represents the comprehensive acceleration of the vehicle after the driving right is allocated at the current moment; ρ v and ρ a Represents adjustment factors, and are all positive numbers; represents the heading angle of the vehicle at the current moment; ω represents the normalized lateral offset distance of the vehicle.

[0035] Furthermore, the reward function for successful lane change in step 4.2 is given by the formula Sure, Indicates that the lane change is successful;

[0036] The efficiency reward function is given by the formula Sure;

[0037] The distance reward function is given by the formula OK, when Indicates that a collision occurs, and the training is terminated;

[0038] The comfort reward function is given by the formula Sure;

[0039] In the formula, Indicates the current speed of the vehicle ahead in the lane; Indicates the current driving speed of the vehicle ahead in the adjacent lane; Indicates the current speed of the vehicle; Indicates the relative lateral position of the vehicle and the vehicle in front of the lane at the current moment; Indicates the relative lateral position of the vehicle and the vehicle behind it in the adjacent lane at the current moment; Indicates the relative lateral position of the vehicle and the preceding vehicle in the adjacent lane at the current moment; Indicates the relative longitudinal position of the vehicle and the vehicle behind it in the adjacent lane at the current moment; Indicates the relative longitudinal position of the vehicle and the preceding vehicle in the adjacent lane at the current moment; according to the three different driving modes of fully automated driving system driving, fully driver model driving, and human-machine co-driving, δ t Corresponding to ρ e , x , y , δ Represents the adjustment factor and is a positive number.

[0040] Furthermore, the lane keeping model fusion reward function in step 4.3 is determined by the following formula:

[0041]

[0042] The left and right lane change model fusion reward function is determined by the following formula:

[0043]

[0044] Where: represents the basic reward function of the lane keeping model in human-machine co-driving; Represents the basic reward function of the lane keeping model in the fully automated driving system driving style; Represents the basic reward function of the lane keeping model in the full driver model approach; Represents the basic reward function of the left-right lane-changing model in the human-machine co-driving mode; Represents the basic reward function for the left and right lane change model in the fully automated driving system driving style; represents the basic reward function of the left and right lane changing model in the full driver model; η1, η2, η3, λ1, λ2, λ3 represent adjustment factors.

[0045] Furthermore, step 5 specifically includes:

[0046] Step 5.1: Establish a driving simulation environment required for training the lane keeping model. The driving simulation environment includes one-way two-lane roads, straight roads, curves, and traffic lights, no traffic flow, and a preset distance from the start point to the end point;

[0047] Step 5.2: Based on step 5.1, set up the scene and establish the driving simulation environment required for training the left and right lane change model;

[0048] Step 5.3: Prioritize the overall environment state space and driving right allocation coefficient action space obtained by the three modes of fully autonomous driving system driving, fully driver model driving and human-machine co-driving, and simultaneously input them into the lane keeping model fusion reward function of step 4.3 The lane keeping model of step 5.1 is trained in a driving simulation environment. If the training fails, the training is repeated until the vehicle safely reaches the destination, and the trained model is saved.

[0049] Step 5.4: Put the model trained in step 5.3 into the left and right lane change model of step 4.3 and fuse the reward function R c 融 Further training is performed in the driving simulation environment of the left and right lane changing model of step 5.2. If the training fails, the training is repeated until the vehicle reaches the destination safely and efficiently while continuously completing the lane changing action, and the driving right allocation strategy of the human-machine shared driving system is obtained, and the allocation weight is output.

[0050] The beneficial effects of the present invention are:

[0051] First, the present invention uses a hierarchical deep reinforcement learning method to solve the problem of continuous dynamic allocation of lateral and longitudinal driving control rights between the driver and the automatic driving system in the human-machine shared driving system of an intelligent vehicle, simplifies the human-machine shared driving system's allocation process of driving control rights, sets multiple training targets, namely the lane keeping model and the left and right lane changing model, performs hierarchical training on the lane keeping model and the left and right lane changing model, preferentially inputs the environmental state space of the lane keeping model and the driving right allocation coefficient action space into the lane keeping model training, and then trains the left and right lane changing model, thereby improving the training efficiency and accuracy, and designs different reward functions for the two subtasks, so that the human-machine shared driving system takes into account the lateral and longitudinal driving weight allocation of the driver and the automatic driving system while ensuring safe driving, thereby improving comfort;

[0052] Second, the present invention adopts a deep deterministic strategy algorithm training model, which can handle the multi-dimensional continuous states and actions in the driving rights allocation problem, and enable the human-machine shared driving system to have stronger exploration capabilities, while improving stability and convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flowchart of a multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning of the present invention;

[0054] Figure 2 is a schematic diagram of a vehicle kinematics model of an embodiment of the present invention;

[0055] Figure 3 is a schematic diagram of a driving simulation environment of a lane keeping model according to an embodiment of the present invention;

[0056] Figure 4 is a schematic diagram of a driving simulation environment of a left-right lane change model according to an embodiment of the present invention;

[0057] Figure 5 It is a schematic diagram of the deep reinforcement learning DDPG algorithm framework of an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the technical solution of the present application, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0059] The terms such as up, down, left, right, front and back in this application document are based on the positional relationships shown in the drawings. If the drawings are different, the corresponding positional relationships may also change accordingly, so they cannot be understood as limiting the scope of protection.

[0060] In this application, the terms "install", "connected", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, or mutual communication, a direct connection, or an indirect connection through an intermediate medium, the internal connection of two components, or the interaction relationship between two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0061] See the instruction manual Figure 1 ,The multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning includes the following steps:

[0062] Step 1: The human-machine shared driving system obtains the environmental observation states of the autonomous driving vehicle in the driving simulation environment through sensor equipment. Each environmental observation state includes at least the vehicle's own state, the vehicle's surrounding environment state and the driver's control input state.

[0063] It should be noted that the sensing equipment in step 1 is installed on the autonomous driving vehicle using existing technology, including vehicle cameras, radars, inertial navigation and other sensing equipment.

[0064] Step 2: Based on the environmental observation states of step 1, establish the overall environmental state space and the vehicle kinematic model that describes the lateral and longitudinal dynamic behavior of the vehicle, and based on the driving rights allocation strategy of the human-vehicle shared driving system, establish the driving rights allocation coefficient action space of the driver and the automatic driving system.

[0065] For details, please refer to the instruction manual. Figure 2 , step 2 specifically includes:

[0066] Step 2.1: The environmental observation states obtained by the sensor equipment in the driving simulation environment of the autonomous driving vehicle are defined as the vehicle's own state space S1, the vehicle's surrounding environment state space S2, and the driver's control input state space S3.

[0067] Step 2.2: Use the geodetic coordinate system, take the center of mass position of the autonomous driving vehicle at the initial moment as the coordinate origin, the direction of moving along the lane line is the lateral direction, and the direction of rotating 90° counterclockwise from the lateral direction is the longitudinal direction. The vehicle is regarded as a rectangular rigid body, and each environmental observation state space is established according to the motion characteristics. The overall environmental state space S = [S1, S2, S3] is established through each environmental observation state space.

[0068] It should be noted that the horizontal direction is the positive direction of the X-axis in the geodetic coordinate system, and the vertical direction is the positive direction of the Y-axis in the geodetic coordinate system.

[0069] Furthermore, the vehicle's own state space in: Indicates the lateral position coordinates of the vehicle at the current moment; Indicates the longitudinal position coordinates of the vehicle at the current moment; Indicates the current speed of the vehicle; Indicates the acceleration input by the self-driving system at the current moment; Indicates the steering angle input by the self-driving system at the current moment; Indicates the comprehensive acceleration of the vehicle after the driving right is allocated at the current moment; Indicates the comprehensive steering angle of the current driving right distribution and the rear vehicle; φ t Indicates the heading angle of the vehicle at the current moment.

[0070] Vehicle surroundings state space in: Indicates the relative lateral position of the vehicle and the vehicle in front of the lane at the current moment; Indicates the relative lateral position of the vehicle and the vehicle behind it in the adjacent lane at the current moment; Indicates the relative lateral position of the vehicle and the preceding vehicle in the adjacent lane at the current moment; Indicates the relative longitudinal position of the vehicle and the vehicle behind it in the adjacent lane at the current moment; Indicates the relative longitudinal position of the vehicle and the preceding vehicle in the adjacent lane at the current moment; V l t Indicates the current speed of the vehicle ahead in the lane; Indicates the current speed of the vehicle ahead in the adjacent lane; L indicates lane line information; T indicates traffic signal information.

[0071] Driver control input state space in: Indicates the steering angle input by the driver through the steering wheel at the current moment; Indicates the acceleration input by the driver through the brake and accelerator at the current moment.

[0072] Step 2.3: The driving right allocation strategy learned by the human-vehicle shared driving system is used to coordinate the driving weights of the driver and the autonomous driving system. The driving right allocation strategy makes the optimal lateral and longitudinal driving right allocation based on the real-time feedback state space and the rewards obtained, and defines the driving right allocation coefficient action space as A.

[0073] Specifically, the driving rights allocation coefficient action space in: It represents the lateral driving right allocation coefficient of the automatic driving system; Indicates the driver's lateral driving right allocation coefficient; It represents the longitudinal driving right allocation coefficient of the automatic driving system; Indicates the driver's longitudinal driving right allocation coefficient; and

[0074] According to the driving right allocation coefficient action space A, the vehicle's own state matrix S1 is obtained

[0075] Step 3: Use hierarchical reinforcement learning to divide the overall environmental state space and driving right allocation coefficient action space of the human-vehicle shared driving system obtained in step 2 into two layers of training targets. The first layer of training targets is the lane keeping model, and the second layer of training targets is the left and right lane changing model.

[0076] Specifically, step 3 includes:

[0077] Step 3.1: Obtain the environment state space S of the lane keeping model through hierarchical reinforcement learning k and driving rights allocation coefficient action space A; S k It is composed of some elements in the overall environment state space S = [S1, S2, S3], The environment state space S of the lane keeping model k a t Include and The driving rights allocation coefficient action space A is the same as step 2.3.

[0078] Step 3.2: Obtain the environment state space S of the left and right lane change model through hierarchical reinforcement learning c and driving rights allocation coefficient action space A; S c It is composed of some elements in the overall environment state space S = [S1, S2, S3], The environmental state space S of the left-right lane-changing model c δ t Include and The driving rights allocation coefficient action space A is the same as step 2.3.

[0079] Step 4: Based on the vehicle kinematic model in step 2 and the lane keeping model and left and right lane changing models decomposed in step 3, basic reward functions are established respectively, and they are integrated according to the three modes of fully automatic driving system driving, full driver model and human-machine co-driving to obtain the lane keeping model fusion reward function and the left and right lane changing model fusion reward function.

[0080] It should be noted that the full driver model is used to simulate the driver's operations during the training process.

[0081] Specifically, step 4 includes:

[0082] Step 4.1: Basic Reward Function R for Lane Keeping Model k By approaching the end point reward function Efficiency Reward Function Lane Keeping Reward Function and the rule-following reward function OK. The formula is as follows:

[0083]

[0084] Furthermore, the reward function near the end point is expressed as Sure.

[0085] The efficiency reward function is given by the formula OK, when When , the training is terminated.

[0086] The lane keeping reward function is given by the formula OK, when When , the training is terminated.

[0087] The rule-following reward function is given by the formula Sure, Terminate training.

[0088] Where: Indicates the lateral position coordinates of the vehicle at the current moment; Indicates the current speed of the vehicle; according to the three different driving modes of fully automatic driving system driving, fully driver model driving and human-machine co-driving, a t Corresponding to ρ v and ρ a Represents adjustment factors, all of which are positive numbers and are adjusted according to training results; represents the heading angle of the vehicle at the current moment; ω represents the normalized lateral offset distance of the vehicle.

[0089] Step 4.2: Basic reward function R for left-right lane change model c Through the successful lane change reward function Efficiency Reward Function Distance Reward Function And the comfort reward function OK. The formula is as follows:

[0090]

[0091] Furthermore, the reward function for successful lane change is given by the formula Sure, Indicates that the lane change was successful.

[0092] The efficiency reward function is given by the formula Sure.

[0093] The distance reward function is given by the formula OK, when Indicates that a collision occurs, and the training is terminated.

[0094] The comfort reward function is given by the formula Sure.

[0095] Where: Indicates the current speed of the vehicle ahead in the lane; Indicates the current driving speed of the vehicle ahead in the adjacent lane; Indicates the current speed of the vehicle; Indicates the relative lateral position of the vehicle and the vehicle in front of the lane at the current moment; Indicates the relative lateral position of the vehicle and the vehicle behind it in the adjacent lane at the current moment; Indicates the relative lateral position of the vehicle and the preceding vehicle in the adjacent lane at the current moment; Indicates the relative longitudinal position of the vehicle and the vehicle behind it in the adjacent lane at the current moment; Indicates the relative longitudinal position of the vehicle and the preceding vehicle in the adjacent lane at the current moment; according to the three different driving modes of fully automated driving system driving, fully driver model driving, and human-machine co-driving, δ t Corresponding to ρ e , x , y , δ Represents adjustment factors, all of which are positive numbers and are adjusted according to training results.

[0096] Step 4.3: Based on the basic reward functions of steps 4.1 and 4.2, the lane keeping model fusion reward function is obtained by integrating the three driving modes: fully automated driving system driving, fully driver model driving, and human-machine co-driving. Fusion reward function with left and right lane change model

[0097] The lane keeping model fusion reward function is determined by the following formula:

[0098]

[0099] The left and right lane change model fusion reward function is determined by the following formula:

[0100]

[0101] Where: Represents the basic reward function of the lane keeping model in human-machine co-driving; Represents the basic reward function of the lane keeping model in the fully automated driving system driving style; Represents the basic reward function of the lane keeping model in the full driver model approach; Represents the basic reward function of the left-right lane-changing model in the human-machine co-driving mode; Represents the basic reward function for the left and right lane change model in the fully automated driving system driving style; represents the basic reward function of the left and right lane changing model in the full driver model; η1, η2, η3, λ1, λ2, λ3 represent adjustment factors.

[0102] Step 5: Based on the lane keeping model fusion reward function and the left and right lane changing model fusion reward function established in step 4, training is performed. At the same time, a driving simulation environment required for the lane keeping model and the left and right lane changing model training is established. The overall environmental state space and driving right allocation coefficient action space obtained by the three modes of fully automatic driving system driving, fully driver model driving and human-machine co-driving are simultaneously input into the driving simulation environment and fusion reward function of the lane keeping model for training. The trained model is input into the driving simulation environment and fusion reward function of the left and right lane changing model for further training until the optimal driving right allocation strategy for the human-machine shared driving system is obtained through training, and the allocation weight is output.

[0103] Specifically, step 5 includes:

[0104] Step 5.1: Establish a driving simulation environment required for training the lane keeping model. The driving simulation environment includes one-way two-lane roads, straight roads, curves and traffic lights, no traffic flow, and a preset distance from the starting point to the end point.

[0105] See the instruction manual Figure 3 , attached Figure 3 This is an illustration of the lane keeping model, where the starting point and the end point of the vehicle are set to be 1000m apart.

[0106] Step 5.2: Based on step 5.1, set up the scene and establish the driving simulation environment required for training the left and right lane change model.

[0107] See the instruction manual Figure 4 , attached Figure 4 The left lane is the driving lane, the right lane is the adjacent lane, and the adjacent lane of the driving lane is where the front vehicle and the rear vehicle are located.

[0108] Step 5.3: Prioritize the overall environment state space and driving right allocation coefficient action space obtained by the three modes of fully autonomous driving system driving, fully driver model driving and human-machine co-driving, and simultaneously input them into the lane keeping model fusion reward function of step 4.3 The lane keeping model of step 5.1 is trained in a driving simulation environment. If the training fails, the training is repeated until the vehicle safely reaches the destination, and the trained model is saved.

[0109] Step 5.4: Put the model trained in step 5.3 into the left and right lane change model fusion reward function of step 4.3 Further training is performed in the driving simulation environment of the left and right lane changing model of step 5.2. If the training fails, the training is repeated until the vehicle reaches the destination safely and efficiently while continuously completing the lane changing action. Finally, the trained model is saved, and the driving right allocation strategy of the human-machine shared driving system is obtained, and the allocation weight is output.

[0110] It should be noted that during training, the autonomous driving system model uses the model predictive control algorithm to track and control the established road, and the output is the steering angle input by the autonomous driving system of the vehicle at the current moment. and the acceleration input by the current self-driving system As one of the inputs of the vehicle's own state space S1 in step 2.2. The full driver model adopts a dual-point preview driver model, with the input being the near preview point and the far preview point, and the output being the steering angle input by the driver through the steering wheel at the current moment. and the acceleration input by the driver through the brake and accelerator at the current moment As the driver control input state space S3 in step 2.2.

[0111] Furthermore, since the overall environment state space and driving right allocation coefficient action space obtained by the human-machine shared driving system have multi-dimensional continuity, a deep reinforcement learning algorithm is used to train the lane keeping model and the left and right lane changing model respectively. The deep reinforcement learning algorithm of this embodiment adopts a common algorithm, taking the deep deterministic policy gradient (DDPG) algorithm as an example, refer to the appendix of the specification. Figure 5 , the training process of step 5.3 and step 5.4 is as follows:

[0112] S1: DDPG uses four neural networks, namely online Actor: μ(S|θ), target Actor: μ'(S|θ'), online Critic: Q(S,A|φ) and target Critic: Q'(S,A|φ'). The policy function is fitted by online Actor: μ(S|θ) and target Actor: μ'(S|θ') to generate the allocation result of human-machine co-driving control rights, and the action value function is fitted by online Critic: Q(S,A|φ) and target Critic: Q'(S,A|φ') to judge the allocation effect.

[0113] S2: Initialize the experience replay pool D, initialize 4 neural networks, and initialize a random noise n t Used to explore the environment, Actor: μ(S|θ) obtains the driving right allocation action a based on the current environment observation state S t , assign driving rights to action a t Add random noise n t , generating an exploratory driving rights allocation action a t =μ(S t |θ)+n t , and get the reward value r t , and transfer the environment to the next environmental observation state S t+1 .

[0114] S3: Create a matrix [(S t ,a t ,r t ),(S t+1 ,a t+1 ,r t+1 ),……,(S t+n ,a t+n ,r t+n )] and store it in the experience replay pool D. In each round of the cycle, N samples are randomly sampled from the experience replay pool D to calculate the target value y i =r i +γQ'(S i+1 ,μ'(S i+1 |θ μ’ )|θ Q’ ), where γ is the learning rate; then the online critic network parameter φ is updated by minimizing the loss: Then use the Monte Carlo method to estimate the policy gradient and update the online Actor network parameters θ: Finally, the parameters of the target Actor network and the Critic network are updated with the parameters θ of the online Actor network and the parameters φ of the online Critic network to obtain the driving rights allocation strategy.

[0115] In this embodiment, the human-vehicle shared driving process is modeled through the Markov decision process (MDP). The Markov decision process is a theoretical framework for studying how intelligent agents learn to achieve specific goals during the interaction with the environment. It is represented by a quaternion [S, π, A, R], where S is the overall environment state space established for each environmental observation state, A is the driving right allocation coefficient action space, R is the human-vehicle shared driving system reward, and π is the driving right allocation strategy.

[0116] In the process of human-machine co-driving, the human-machine shared driving system, as an intelligent agent, obtains the environmental information required to allocate driving control rights by observing the environmental state space S, and then makes the driving right allocation coefficient action space A according to the driving right allocation strategy π. After receiving A, the environment feedbacks the human-vehicle shared driving system reward R and enters the next environmental state space S t+1 , and the following sequence is formed by repeated steps: [S0, A0, S1, A1, R1, S2, A2, R2, …, S t+1 ,A t+1 ,R t+1 ]. The goal of human-machine co-driving is to obtain rewards through the interaction between the human-machine shared driving system and the driving environment, and to maximize the expected total return of the cumulative rewards. Finally, a driving control rights allocation strategy that meets the expected effect is trained, and the allocation weights are output through the final driving control rights allocation strategy.

[0117] The above is only an embodiment of the present invention, and the common sense such as the known specific structure and characteristics in the scheme is not described in detail here. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the attached claims rather than the above description, so it is intended to include all changes within the meaning and scope of the equivalent elements of the claims. Any figure mark in the claims should not be regarded as limiting the claims involved.

Claims

1. A multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning, characterized in that: The multi-objective vehicle-sharing control method based on hierarchical deep reinforcement learning comprises the following steps: Step 1: The human-machine shared driving system obtains various environmental observation states of the autonomous driving vehicle in the driving simulation environment through sensor equipment. Each environmental observation state at least includes the vehicle's own state, the vehicle's surrounding environment state, and the driver's control input state; Step 2: Based on the environmental observation states in step 1, establish the overall environmental state space and the vehicle kinematic model that describes the lateral and longitudinal dynamic behaviors of the vehicle, and based on the driving rights allocation strategy of the human-vehicle shared driving system, establish the driving rights allocation coefficient action space of the driver and the automatic driving system; Step 3: Use hierarchical reinforcement learning to divide the overall environmental state space and driving right allocation coefficient action space of the human-vehicle shared driving system obtained in step 2 into two layers of training targets. The first layer of training targets is the lane keeping model, and the second layer of training targets is the left and right lane changing model. Step 4: Based on the vehicle kinematic model in step 2 and the lane keeping model and left and right lane changing models decomposed in step 3, basic reward functions are established respectively, and they are integrated according to the three modes of fully automated driving system driving, full driver model and human-machine co-driving to obtain the lane keeping model fusion reward function and the left and right lane changing model fusion reward function; Step 5: Based on the lane keeping model fusion reward function and the left and right lane changing model fusion reward function established in step 4, training is performed. At the same time, a driving simulation environment required for the lane keeping model and the left and right lane changing model training is established. The overall environmental state space and driving right allocation coefficient action space obtained by the three modes of fully automatic driving system driving, fully driver model driving and human-machine co-driving are simultaneously input into the driving simulation environment and fusion reward function of the lane keeping model for training. The trained model is input into the driving simulation environment and fusion reward function of the left and right lane changing model for further training until the optimal driving right allocation strategy for the human-machine shared driving system is obtained through training, and the allocation weight is output.

2. The multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning according to claim 1 is characterized in that: In step 2, the overall environment state space S = [S1, S2, S3] is established based on the environment observation states in step 1; Where: S1 is the state space of the vehicle itself; S2 is the state space of the vehicle's surrounding environment; S3 is the driver's control input state space; In step 2, the driving rights allocation strategy of the human-vehicle shared driving system is used to establish the driving rights allocation coefficient action space between the driver and the automatic driving system. Where: It represents the lateral driving right allocation coefficient of the automatic driving system; Indicates the driver's lateral driving right allocation coefficient; It represents the longitudinal driving right allocation coefficient of the automatic driving system; Indicates the driver's longitudinal driving right allocation coefficient; and 3. The multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning according to claim 2 is characterized in that: Vehicle state space Vehicle surroundings state space Driver control input state space Where: Indicates the lateral position coordinates of the vehicle at the current moment; Indicates the longitudinal position coordinates of the vehicle at the current moment; Indicates the current speed of the vehicle; Indicates the acceleration input by the self-driving system at the current moment; Indicates the steering angle input by the self-driving system at the current moment; Indicates the comprehensive acceleration of the vehicle after the driving right is allocated at the current moment; Indicates the comprehensive steering angle of the current driving right distribution and the rear vehicle; φ t Indicates the heading angle of the vehicle at the current moment; Indicates the relative lateral position of the vehicle and the vehicle in front of the lane at the current moment; Indicates the relative lateral position of the vehicle and the vehicle behind it in the adjacent lane at the current moment; Indicates the relative lateral position of the vehicle and the preceding vehicle in the adjacent lane at the current moment; Indicates the relative longitudinal position of the vehicle and the vehicle behind it in the adjacent lane at the current moment; Indicates the relative longitudinal position of the vehicle and the preceding vehicle in the adjacent lane at the current moment; Indicates the current speed of the vehicle ahead in the lane; Indicates the current driving speed of the vehicle ahead in the adjacent lane; L indicates lane line information; T indicates traffic signal information; Indicates the steering angle input by the driver through the steering wheel at the current moment; Indicates the acceleration input by the driver through the brake and accelerator at the current moment.

4. The multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning according to claim 3 is characterized in that: The lane keeping model in step 3 contains the environment state space S k and driving rights allocation coefficient action space A, The environment state space S of the lane keeping model k a t Include and The left-right lane-changing model in step 3 contains the environment state space S c and driving rights allocation coefficient action space A, The environmental state space S of the left-right lane-changing model c δ t Include and In step 3, the driving right allocation coefficient action space A of the lane keeping model is the same as the driving right allocation coefficient action space A of the left-right lane changing model.

5. The multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning according to claim 3 is characterized in that: Step 4 specifically includes: Step 4.1: Establish the basic reward function R of the lane keeping model k , R k Determined by the following formula: Where: Represents the reward function for approaching the end point; represents the efficiency reward function; represents the lane keeping reward function; represents the reward function for following the rules; Step 4.2: Establish the basic reward function R for the left and right lane change model c , R c Determined by the following formula: Where: represents the reward function for successful lane change; represents the efficiency reward function; represents the distance reward function; represents the comfort reward function; Step 4.3: Based on the basic reward functions of steps 4.1 and 4.2, the lane keeping model fusion reward function is obtained by integrating the three driving modes: fully automated driving system driving, fully driver model driving, and human-machine co-driving. Fusion reward function with left and right lane change model 6. The multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning according to claim 5 is characterized in that: Reward function for approaching the end point in step 4.1 By formula Sure; The efficiency reward function is given by the formula OK, when When , the training is terminated; The lane keeping reward function is given by the formula OK, when When , the training is terminated; The rule-following reward function is given by the formula Sure, Terminate training; In the formula, Indicates the current speed of the vehicle; according to three different driving modes: fully automatic driving system driving, fully driver model driving, and human-machine co-driving; a t Corresponding to Indicates the acceleration input by the self-driving system at the current moment; Indicates the steering angle input by the self-driving system at the current moment; Indicates the comprehensive acceleration of the vehicle after the driving right is allocated at the current moment; ρ v and ρ a Represents adjustment factors, and are all positive numbers; represents the heading angle of the vehicle at the current moment; ω represents the normalized lateral offset distance of the vehicle.

7. The multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning according to claim 5 is characterized in that: The reward function for successful lane change in step 4.2 is given by the formula Sure, Indicates that the lane change is successful; The efficiency reward function is given by the formula Sure; The distance reward function is given by the formula OK, when Indicates that if a collision occurs, the training is terminated; The comfort reward function is given by the formula Sure; In the formula, Indicates the current speed of the vehicle ahead in the lane; Indicates the current driving speed of the vehicle ahead in the adjacent lane; Indicates the current speed of the vehicle; Indicates the relative lateral position of the vehicle and the vehicle in front of the lane at the current moment; Indicates the relative lateral position of the vehicle and the vehicle behind it in the adjacent lane at the current moment; Indicates the relative lateral position of the vehicle and the preceding vehicle in the adjacent lane at the current moment; Indicates the relative longitudinal position of the vehicle and the vehicle behind it in the adjacent lane at the current moment; Indicates the relative longitudinal position of the vehicle and the preceding vehicle in the adjacent lane at the current moment; according to the three different driving modes of fully automated driving system driving, fully driver model driving, and human-machine co-driving, δ t Corresponding to ρ e , x , y , δ Represents the adjustment factor and is a positive number.

8. The multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning according to claim 5 is characterized in that: The lane keeping model fusion reward function in step 4.3 is determined by the following formula: The left and right lane change model fusion reward function is determined by the following formula: Where: represents the basic reward function of the lane keeping model in human-machine co-driving; Represents the basic reward function of the lane keeping model in the fully automated driving system driving style; Represents the basic reward function of the lane keeping model in the full driver model approach; Represents the basic reward function of the left-right lane-changing model in the human-machine co-driving mode; Represents the basic reward function for the left and right lane change model in the fully automated driving system driving style; represents the basic reward function of the left and right lane changing model in the full driver model; η1, η2, η3, λ1, λ2, λ3 represent adjustment factors.

9. The multi-objective human-vehicle sharing control method based on hierarchical deep reinforcement learning according to claim 8 is characterized in that: Step 5 specifically includes: Step 5.1: Establish a driving simulation environment required for training the lane keeping model. The driving simulation environment includes one-way two-lane roads, straight roads, curves, and traffic lights, no traffic flow, and a preset distance from the start point to the end point; Step 5.2: Based on step 5.1, set up the scene and establish the driving simulation environment required for training the left and right lane change model; Step 5.3: Prioritize the overall environment state space and driving right allocation coefficient action space obtained by the three modes of fully autonomous driving system driving, fully driver model driving and human-machine co-driving, and simultaneously input them into the lane keeping model fusion reward function of step 4.3 The lane keeping model of step 5.1 is trained in a driving simulation environment. If the training fails, the training is repeated until the vehicle safely reaches the destination, and the trained model is saved. Step 5.4: Put the model trained in step 5.3 into the left and right lane change model fusion reward function of step 4.3 Further training is performed in the driving simulation environment of the left and right lane changing model of step 5.

2. If the training fails, the training is repeated until the vehicle reaches the destination safely and efficiently while continuously completing the lane changing action, and the driving right allocation strategy of the human-machine shared driving system is obtained, and the allocation weight is output.

Images