Reinforcement learning and predictive control-based elk test driver modeling method and system
By using driver modeling methods based on strengthening learning and predicting control in the elk test, the problem of lack of precision driver models in the simulation of the car closed-loop system of the car, the more accurate and objective test results are achieved, the test efficiency is improved, and the test efficiency is improved. And security.
Patent Information
- Application Number
- CN202411198431.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2025-01-28
AI Technical Summary
In the elk test, the human-vehicle closed-loop system simulation lacks an accurate driver model, which leads to a deviation from the actual driving situation, and cannot truly reflect the performance of the vehicle in actual driving.
Adopt the Modk test driver modeling method based on strengthening learning and predicting control, to build a smart body of the learning algorithm by constructing a deepening -to -see -based strategy gradient, to obtain the feature value and speed of testing roads, output optimal driving path parameters, and predict control through model prediction control through model prediction control The algorithm tracks the trajectory to modify the smart parameters until the training is completed.
Achieve precise simulation of driver behavior in the elk test, eliminate the impact of human factors, ensure the objectivity and accuracy of the test results, improve the accuracy of the vehicle control, ensure the safety of the test process, and improve the test efficiency of testing efficiency. Essence
Smart Images
Figure CN119376272A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of vehicle testing, and in particular relates to a modeling method and system for a moose test driver based on reinforcement learning and predictive control. Background Art
[0002] The Moose Test, also known as the emergency avoidance test, is an important international standard for measuring vehicle safety. Its name comes from the moose (also known as elk in some areas) that are common in Scandinavia and North America. These animals often jump out in front of the vehicle unexpectedly and collide with high-speed vehicles, causing serious traffic accidents. Therefore, the Moose Test is designed to test the vehicle's ability to avoid obstacles while driving at high speeds. This test is widely used by the global automotive industry. During the test, the vehicle enters the test area at a constant speed and dodges at high speed without braking or accelerating. The tester will repeat this test and gradually increase the speed until the vehicle loses control, and finally determine the reliability of the out-of-control speed value.
[0003] In the moose test, the core of the closed-loop human-vehicle system simulation is to simulate the operation behavior of a real driver in an emergency obstacle avoidance situation, as well as the vehicle's response to this operation. The driver model is a key part of the simulation system to simulate the driver's decision-making and operation. However, the simulation process usually lacks an accurate driver model. The system will not be able to accurately simulate the driver's reaction speed, judgment ability and operating habits in an emergency, resulting in a deviation between the simulation results and the actual driving situation, and unable to truly reflect the performance of the vehicle in actual driving. At the same time, inaccurate simulation results may make it difficult for automakers to accurately judge the shortcomings of vehicle performance and the direction that needs to be optimized, thereby affecting the continuous improvement and upgrading of products. Summary of the invention
[0004] In order to solve the problem of lack of driver model in the simulation of human-vehicle closed-loop system in the moose test, the present invention proposes a moose test driver modeling method and system based on reinforcement learning and predictive control.
[0005] A method for modeling a elk test driver based on reinforcement learning and predictive control to achieve one of the purposes of the present invention comprises the following steps:
[0006] S1. Build an intelligent agent based on deep deterministic policy gradient reinforcement learning algorithm;
[0007] S2, obtaining the test road characteristic value and vehicle speed at each moment during the elk test simulation; inputting the test road characteristic value and vehicle speed into the intelligent agent to obtain the optimal driving path parameter at the next moment; inputting the optimal driving path parameter at the next moment into the model predictive control algorithm for trajectory tracking to obtain the deviation value of the vehicle deviating from the trajectory; and correcting the intelligent agent according to the deviation value;
[0008] The above step S2 is repeatedly executed until a trained intelligent agent is obtained, wherein the trained intelligent agent is a driver model for the elk test.
[0009] Furthermore, in the above step S2, the intelligent agent outputs the action space to the path generator according to the state space and the reward value to obtain the optimal driving path parameters at the next moment; the predictive control algorithm based on the vehicle dynamics model performs vehicle dynamics simulation prediction according to the optimal driving path parameters at the next moment to obtain the deviation value of the vehicle deviation from the trajectory, and inputs the deviation value of the vehicle deviation from the trajectory into the reward function of the intelligent agent to obtain the reward value; the intelligent agent is corrected according to the reward value to minimize the deviation value.
[0010] Furthermore, the deep deterministic policy gradient reinforcement learning algorithm includes a policy network, which is used to generate an action space according to the state space of the virtual environment input and the set learning weights, that is, the optimal driving path parameters at the next moment, wherein the state space includes the speed of the vehicle when the accelerator is released, the width and length of multiple test roads, and the center position of the lane; the action space includes [s1, s2, s3, c1, c2, y c1 ,y c2 ]. s1, s2, s3 represent the straight-line distance in the straight-line driving area, the straight-line distance in the lane-changing driving area, and the straight-line distance after the vehicle returns to the original lane in the elk test; c1 and c2 are the longitudinal lengths of the first and second curves of the trajectory, respectively; y c1 ,y c2 The lateral distance that the lane change area deviates from the center line and the lateral distance that the vehicle deviates from the center line after returning to the original lane area.
[0011] Furthermore, the calculation formula of the strategy network includes:
[0012] a=μ(s;θ μ )
[0013] In the formula, a represents the action space, s represents the state space, and θ μ Represents the learning weights of the policy network.
[0014] Furthermore, the deep deterministic policy gradient reinforcement learning algorithm includes a value network, which is used to output a reward value based on the dynamic characteristic parameters and the maximum number of iteration steps of the vehicle when it travels along the planned path during the moose test. The dynamic characteristic parameters include: longitudinal slip rate of the front and rear axles, lateral slip rate of the front and rear axles, the distance between the center of mass of the vehicle and the nearest point on the path, and yaw angle error.
[0015] Furthermore, the calculation formula of the value network includes:
[0016]
[0017] Where q is the reward value output by the value network; κ fx and κ rx are the longitudinal slip rates of the front and rear axles, respectively; fy and κ ry are the lateral slip rates of the front and rear axles respectively; D v is the distance between the center of mass of the vehicle and the nearest point on the path, that is, the lateral distance that the vehicle deviates from the planned path; Δψ v is the yaw angle error N d,max is the maximum number of iteration steps.
[0018] A simulation system for elk testing based on reinforcement learning and predictive control to achieve the second objective of the present invention comprises:
[0019] Agent building module: used to build an agent based on the deep deterministic policy gradient reinforcement learning algorithm;
[0020] Agent training module: obtain the test road characteristic value and vehicle speed at each moment during the elk test simulation; input the test road characteristic value and vehicle speed into the agent to obtain the optimal driving path parameters at the next moment; input the optimal driving path parameters at the next moment into the model predictive control algorithm for trajectory tracking to obtain the deviation value of the vehicle from the trajectory; and correct the agent according to the deviation value;
[0021] Driver model acquisition module: used to acquire the driver model of the elk test; the acquisition method includes: repeatedly training the intelligent agent until the number of iterations is reached.
[0022] Furthermore, the deep deterministic policy gradient reinforcement learning algorithm includes a policy network, which is used to generate an action space according to the state space of the virtual environment input and the set learning weights, wherein the state space includes the speed of the vehicle when the accelerator is released, the width and length of multiple test roads, and the center position of the lane; the action space includes [s1, s2, s3, c1, c2, y c1 ,y c2], s1, s2, s3 represent the straight-line distance in the straight-line driving area, the straight-line distance in the lane-changing driving area, and the straight-line distance after the vehicle returns to the original lane in the elk test; c1 and c2 are the longitudinal lengths of the first and second curves of the trajectory, respectively; y c1 ,y c2 The lateral distance that the lane change area deviates from the center line and the lateral distance that the vehicle deviates from the center line after returning to the original lane area.
[0023] Furthermore, the deep deterministic policy gradient reinforcement learning algorithm includes a value network, which is used to output a reward value based on the dynamic characteristic parameters and the maximum number of iteration steps when the vehicle travels along the planned path during the moose test.
[0024] The beneficial effects of the present invention include:
[0025] The present invention simulates a driver to perform an elk test through an intelligent agent and a deep deterministic policy gradient (DDPG) reinforcement learning algorithm; tracks the trajectory through an MPC algorithm, calculates the path deviation, and corrects the parameters of the intelligent agent according to the path deviation; the intelligent agent of the present invention can simulate the driver in the elk test, eliminate the influence of human factors on the test results, such as the driver's driving habits, reaction speed, etc., thereby ensuring the objectivity and accuracy of the test results; at the same time, the vehicle can be controlled more accurately to avoid vehicle loss of control or accidents caused by human operating errors, thereby ensuring the safety of the test process; the intelligent agent can perform tests according to preset parameters and conditions to ensure that the conditions of each test are consistent, which helps to more accurately evaluate and compare vehicle performance; and greatly improves the efficiency and speed of the test. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 is a schematic diagram of the method of the present invention;
[0027] Figure 2 is a schematic diagram of the deep deterministic policy gradient reinforcement learning algorithm of the present invention;
[0028] Figure 3 It is a schematic diagram of the simulation environment of the elk experiment;
[0029] Figure 4 It is a parameter diagram of the action space output by the agent;
[0030] Figure 5 It is a schematic diagram of the accurate nonlinear single-track vehicle dynamics model. DETAILED DESCRIPTION
[0031] The following specific implementations are used to explain the technical solutions of the claims of the present invention so that those skilled in the art can understand the claims. The protection scope of the present invention is not limited to the following specific implementation structures. The technical solutions of the claims of the present invention made by those skilled in the art but different from the following specific implementations are also within the protection scope of the present invention.
[0032] The present invention proposes a driver model based on reinforcement learning and predictive control algorithm to complete the elk test simulation.
[0033] like Figure 1 and Figure 2 As shown, Figure 1 The test road and vehicle dynamics model in Figure 2 The environment shown in the figure; the agent outputs the action space to the path generator according to the state space and the reward value. The path generator obtains the parameters of the vehicle path according to the input action space, and then tracks the trajectory through the model predictive control algorithm (such as the MPC algorithm). The reward function calculates the reward value according to the deviation of the vehicle from the trajectory and inputs it to the agent.
[0034] In the absence of training data, reinforcement learning generates its own experience through trial and error. The learning and decision-making algorithm is called an agent; the agent used in the present invention uses the Deep Deterministic Policy Gradient (DDPG) reinforcement learning algorithm, which includes a policy network and a value network. The policy network is used to generate an action space based on the state space of the environment input and the set learning weights. The basic configuration of the algorithm is as follows Figure 2 As shown, the calculation formula is as follows (1):
[0035] a=μ(s;θ μ ) (1)
[0036] In the formula, a represents the action space, s represents the state space, and θ μ represents the learning weight of the policy network; μ(·) represents the target action. The policy network gives a deterministic approximation of the optimal strategy in the action space according to formula (1).
[0037] The simulation environment of the elk test is the road area that the vehicle passes through and the initial speed of the vehicle. The state space s = [v0, l1, w1, x2, y2, l2, w2, x3, y3, l3, w3]; v0 is the speed of the vehicle when the accelerator is released; l1, l2, l3, w1, w2, w3 are the width and length of the three test roads respectively; x2, y2, x3, y3 are the center positions of the lanes of the three test roads respectively, such as Figure 3 shown.
[0038] The action space a is used to generate the vehicle driving path, action space a = [s1, s2, s3, c1, c2, y c1 ,y c2 ],like Figure 4 As shown in the figure, s1, s2, s3 represent the straight-line distance in the straight-line driving area, the straight-line distance in the lane-changing driving area, and the straight-line distance after the vehicle returns to the original lane in the elk test; c1 and c2 represent the longitudinal distances of the first and second curves of the trajectory, respectively; y c1 ,y c2 They respectively represent the lateral distance of the center line of the lane change area and the lateral distance of the center line of the vehicle after returning to the original lane area;
[0039] The value network is expressed as the following formula (2), which is used to output a reward value according to the dynamic characteristic parameters and the maximum number of iteration steps when the vehicle travels along the planned path during the elk test;
[0040]
[0041] Where q is the reward value output by the value network; κ fx and κ rx are the longitudinal slip rates of the front and rear axles respectively; μ fy and κ ry are the lateral slip rates of the front and rear axles respectively; D v is the distance between the center of mass of the vehicle and the nearest point on the path; Δψ v is the maximum number of iterations for the yaw angle error.
[0042] In the elk test simulation, the reward function Q is to quantify the quality of the planned path. The better the planned path, the higher the reward value q. The reward value q is determined by the dynamic characteristics of the vehicle dynamics model when driving along the planned path. In the following cases, the reward value will be negative:
[0043] The longitudinal slip rate of the front and rear axles is ≥ 0.2;
[0044] The lateral slip rate of the front and rear axles is ≥ 0.15;
[0045] The distance between the vehicle's center of mass and the nearest point on the path is ≥ 3 meters;
[0046] The yaw angle error is ≥10°;
[0047] The maximum number of iterations is ≥ 2500.
[0048] The reward value q is calculated as follows:
[0049]
[0050] The parameters are shown in Table 1 below. The score of each parameter is shown in Table 1 below. Substituting the score of each parameter into the above formula (3), the final reward function value can be calculated. An example is as follows:
[0051] Current shaft longitudinal slip rate κ fx When it is greater than or equal to 0.4, the corresponding score is -2;
[0052] When the yaw angle error Δψ v When it is greater than or equal to 15°, the corresponding score is -2.
[0053] Table 1
[0054]
[0055]
[0056] The policy network updates the action space a through the deep deterministic policy gradient (DDPG) reinforcement learning algorithm, and the value network updates the reward value q through the gradient calculated by the temporal difference (TD) error.
[0057] The deep deterministic policy gradient (DDPG) reinforcement learning algorithm process includes:
[0058] 1. Randomly initialize θ Q and θ μ , thereby randomly initializing the value network and the policy network; randomly initializing θ Q' and θ μ' Initialize the target networks Q' and u' of the value network and the policy network; initialize the buffer R;
[0059] 2. Repeat steps 3 to 12 below until the set first iteration number is reached;
[0060] 3. Initialize the random process N and the initial state space S1;
[0061] 4. Repeat steps 5 to 12 below until the second set number of iterations is reached and then return to step 3;
[0062] 5. Select action a based on the current strategy and exploration noise t =μ(s t |θ μ )+N t ;
[0063] 6. Execute action a t , get the reward value r t and the next state s t+1 ;
[0064] 7. Process data (s t ,at ,r t ,s t+1 ) is stored in buffer R;
[0065] 8. Randomly extract M small batches of random samples from the buffer R (s i ,ai,r i ,s i+1 );
[0066] 9. Calculate y according to the following formula i :
[0067] y i =r i +γQ'(s i+1 ,μ'(s i+1 |Q μ′ )|θ Q′ );
[0068] 10. By minimizing the loss function Update the parameters of the value network;
[0069] 11. Using sampled policy gradients Update the parameters of the policy network to optimize the action space parameters of the policy network input;
[0070]
[0071] 12. Update the weights θ of the target network of the value network and the policy network according to the following formula Q′ and θ μ′ ;
[0072] τθ Q +(1-τ)θ Q′ →θ Q′
[0073] τθ μ +(1-τ)θ μ′ →θ μ′
[0074] τ represents the learning rate, τ∈(0,1) is usually set to 0.005.
[0075] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0076] The vehicle dynamics model parameters have the following meanings: Figure 5 As shown, including: State variables: Input: u=δ f; Output: y = Y; its parameters are shown in Table 2 below;
[0077] Table 2
[0078]
[0079]
[0080] The vehicle is in the inertial coordinate system:
[0081]
[0082] The vehicle dynamics model is expressed as:
[0083]
[0084] The continuous state space form of the vehicle dynamics model is obtained:
[0085]
[0086] y=C c x
[0087] in:
[0088]
[0089] B c =[0 b2 b3 0] T
[0090] C c =[1 0 0 0]
[0091]
[0092] The model predictive control algorithm MPC adopted by the present invention is as follows:
[0093] Predictive control is generally a linear discrete system, and the expression is:
[0094] x(k+1)=Ax9k)+Bu(k) (4)
[0095] y(k)=C y x(k) (5)
[0096] Where x is the state variable of the vehicle dynamics model, u is the control input, y is the measured output, and z is the controlled output.
[0097] In the absence of noise and the state variables are obtained through simulation measurements, that is, C y = I. The following formula is obtained by iterating formulas (4) to (5).
[0098]
[0099] The calculation formula of the controlled output z is as follows (9):
[0100] Z(k)=Ψx(k)+Υu(k-1)+ΘΔU(k) (9)
[0101] in:
[0102]
[0103] The embodiment of the present invention also provides a method for modeling a moose test driver based on reinforcement learning and predictive control, comprising:
[0104] S1. Build an intelligent agent based on deep deterministic policy gradient reinforcement learning algorithm;
[0105] S2, obtaining the test road characteristic value and vehicle speed at each moment during the elk test simulation; inputting the test road characteristic value and vehicle speed into the intelligent agent to obtain the optimal driving path parameter at the next moment; inputting the optimal driving path parameter at the next moment into the model predictive control algorithm (MPC) for trajectory tracking to obtain the deviation value of the vehicle deviating from the trajectory; and correcting the intelligent agent according to the deviation value;
[0106] The above step S2 is repeatedly executed until a trained intelligent agent is obtained, wherein the trained intelligent agent is a driver model for the elk test.
[0107] In some embodiments, the deep deterministic policy gradient reinforcement learning algorithm includes a policy network, which is used to generate an action space according to the state space of the virtual environment input and the set learning weights, wherein the state space includes the speed of the vehicle when the accelerator is released, the width and length of multiple test roads, and the center position of the lane; the action space includes [s1, s2, s3, c1, c2, y c1 ,y c2 ], s1, s2, s3 represent the straight-line distance in the straight-line driving area, the straight-line distance in the lane-changing driving area, and the straight-line distance after the vehicle returns to the original lane in the elk test; c1 and c2 are the longitudinal lengths of the first and second curves of the trajectory, respectively; y c1 ,y c2 The lateral distance that the lane change area deviates from the center line and the lateral distance that the vehicle deviates from the center line after returning to the original lane area.
[0108] In some embodiments, the calculation formula of the policy network includes:
[0109] a=μ(s;θ μ )
[0110] In the formula, a represents the action space, s represents the state space, and θ μ Represents the learning weights of the policy network.
[0111] In some embodiments, the deep deterministic policy gradient reinforcement learning algorithm includes a value network, which is used to output a reward value based on the dynamic characteristic parameters and the maximum number of iteration steps when the vehicle travels along the planned path during the moose test.
[0112] The embodiment of the present invention further provides an elk test simulation system based on reinforcement learning and predictive control, comprising:
[0113] Agent building module: used to build an agent based on the deep deterministic policy gradient reinforcement learning algorithm;
[0114] Agent training module: obtain the test road characteristic value and vehicle speed at each moment during the elk test simulation; input the test road characteristic value and vehicle speed into the agent to obtain the optimal driving path parameters at the next moment; input the optimal driving path parameters at the next moment into the model predictive control algorithm for trajectory tracking to obtain the deviation value of the vehicle from the trajectory; and correct the agent according to the deviation value;
[0115] Driver model acquisition module: used to acquire the driver model of the elk test; the acquisition method includes: repeatedly training the intelligent agent until the number of iterations is reached.
[0116] In some embodiments, the deep deterministic policy gradient reinforcement learning algorithm includes a policy network, which is used to generate an action space according to the state space of the virtual environment input and the set learning weights, wherein the state space includes the speed of the vehicle when the accelerator is released, the width and length of multiple test roads, and the center position of the lane; the action space includes [s1, s2, s3, c1, c2, y c1 ,y c2 ], s1, s2, s3 represent the straight-line distance in the straight-line driving area, the straight-line distance in the lane-changing driving area, and the straight-line distance after the vehicle returns to the original lane in the elk test; c1 and c2 are the longitudinal lengths of the first and second curves of the trajectory, respectively; y c1 ,y c2 The lateral distance that the lane change area deviates from the center line and the lateral distance that the vehicle deviates from the center line after returning to the original lane area.
[0117] In some embodiments, the deep deterministic policy gradient reinforcement learning algorithm includes a value network, which is used to output a reward value based on the dynamic characteristic parameters and the maximum number of iteration steps when the vehicle travels along the planned path during the moose test.
[0118] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the steps of the method described in the present invention are implemented, which will not be repeated here.
[0119] The computer-readable storage medium may be the data transmission device provided in any of the aforementioned embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smartmedia card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc., provided on the computer device.
[0120] Furthermore, the computer-readable storage medium may include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data to be output or has been output.
[0121] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0122] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0125] An embodiment of the present invention also provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, performs any step of the moose test driver modeling method based on reinforcement learning and predictive control.
[0126] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.
Claims
1. A modeling method for a moose test driver based on reinforcement learning and predictive control, characterized in that: include: S1. Build an intelligent agent based on deep deterministic policy gradient reinforcement learning algorithm; S2, obtaining the test road characteristic value and vehicle speed at each moment during the elk test simulation; inputting the test road characteristic value and vehicle speed into the intelligent agent to obtain the optimal driving path parameter at the next moment; inputting the optimal driving path parameter at the next moment into the model predictive control algorithm for trajectory tracking to obtain the deviation value of the vehicle deviating from the trajectory; and correcting the intelligent agent according to the deviation value; The above step S2 is repeatedly executed until a trained intelligent agent is obtained, wherein the trained intelligent agent is a driver model for the elk test.
2. The elk test driver modeling method based on reinforcement learning and predictive control as claimed in claim 1, characterized in that: The deep deterministic policy gradient reinforcement learning algorithm includes a policy network, which is used to generate an action space according to the state space of the virtual environment input and the set learning weights, wherein the state space includes the speed of the vehicle when the accelerator is released, the width and length of multiple test roads, and the center position of the lane; the action space includes [s1, s2, s3, c1, c2, y c1 ,y c2 ], s1, s2, s3 represent the straight-line distance in the straight-line driving area, the straight-line distance in the lane-changing driving area, and the straight-line distance after the vehicle returns to the original lane in the elk test; c1 and c2 are the longitudinal lengths of the first and second curves in the elk test trajectory; y c1 ,y c2 The lateral distance that the lane change area deviates from the center line and the lateral distance that the vehicle deviates from the center line after returning to the original lane area.
3. The elk test driver modeling method based on reinforcement learning and predictive control as claimed in claim 2, characterized in that: The calculation formula of the policy network includes: a=μ(s;θ μ ) In the formula, a represents the action space, s represents the state space, and θ μ Represents the learning weights of the policy network.
4. The elk test driver modeling method based on reinforcement learning and predictive control as claimed in claim 2, characterized in that: The deep deterministic policy gradient reinforcement learning algorithm includes a value network, which is used to output a reward value according to the dynamic characteristic parameters and the maximum number of iteration steps when the vehicle travels along the planned path during the moose test.
5. The elk test driver modeling method based on reinforcement learning and predictive control as claimed in claim 4, characterized in that: The calculation formula of the value network includes: Where q is the reward value output by the value network; κ fx and κ rx are the longitudinal slip rates of the front and rear axles, respectively; fy and κ ry are the lateral slip rates of the front and rear axles respectively; D v is the distance between the center of mass of the vehicle and the nearest point on the path; Δψ v is the maximum number of iterations for the yaw angle error.
6. A modeling system for elk test drivers based on reinforcement learning and predictive control, characterized in that: include: Agent building module: used to build an agent based on the deep deterministic policy gradient reinforcement learning algorithm; Agent training module: obtain the test road characteristic value and vehicle speed at each moment during the elk test simulation; input the test road characteristic value and vehicle speed into the agent to obtain the optimal driving path parameters at the next moment; input the optimal driving path parameters at the next moment into the model predictive control algorithm for trajectory tracking to obtain the deviation value of the vehicle from the trajectory; Correcting the agent according to the deviation value; Driver model acquisition module: used to acquire the driver model of the elk test; the acquisition method includes: repeatedly training the intelligent agent until the number of iterations is reached.
7. The elk test driver modeling system based on reinforcement learning and predictive control as claimed in claim 6, characterized in that: The deep deterministic policy gradient reinforcement learning algorithm includes a policy network, which is used to generate an action space according to the state space of the virtual environment input and the set learning weights, wherein the state space includes the speed of the vehicle when the accelerator is released, the width and length of multiple test roads, and the center position of the lane; the action space includes [s1, s2, s3, c1, c2, y c1 ,y c2 ], s1, s2, s3 represent the straight-line distance in the straight-line driving area, the straight-line distance in the lane-changing driving area, and the straight-line distance after the vehicle returns to the original lane in the elk test; c1 and c2 are the longitudinal lengths of the first and second curves in the elk test trajectory; y c1 ,y c2 The lateral distance that the lane change area deviates from the center line and the lateral distance that the vehicle deviates from the center line after returning to the original lane area.
8. The elk test driver modeling system based on reinforcement learning and predictive control as claimed in claim 6 or 7, characterized in that: The deep deterministic policy gradient reinforcement learning algorithm includes a value network, which is used to output a reward value according to the dynamic characteristic parameters and the maximum number of iteration steps when the vehicle travels along the planned path during the moose test.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the moose test driver modeling method based on reinforcement learning and predictive control as described in any one of claims 1 to 5 are implemented.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the moose test driver modeling method based on reinforcement learning and predictive control described in claims 1 to 5 are implemented.