PID trajectory tracking control method and device based on reinforcement learning

Abstract

The invention relates to the field of automatic driving, in particular to a PID trajectory tracking control method and device based on reinforcement learning, and the method comprises the steps: obtaining a current state, a reference trajectory and an actual operation error of a to-be-controlled automatic driving vehicle; constructing a strategy model of vehicle trajectory tracking control; inputting the current state and the reference trajectory into a vehicle trajectory tracking control strategy model to obtain PID control parameters; and inputting the PID control parameter and the actual operation error into a PID controller to obtain the control quantity of the automatic driving vehicle, and completing the trajectory tracking control of the automatic driving vehicle according to the control quantity of the automatic driving vehicle. According to the method, high-real-time, high-stability and high-precision online calculation of vehicle trajectory tracking control is realized by combining the strong learning capability of reinforcement learning with the stability and strong robustness of PID. The problems of poor calculation real-time performance, weak application reliability, dependence on a complex dynamic model and the like of current vehicle trajectory tracking control are solved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a PID trajectory tracking control method and device based on reinforcement learning. Background Art

[0002] Motion control is the core technology in the field of autonomous driving, which is mainly divided into two parts: longitudinal control and lateral control. Among them, longitudinal control refers to the fine adjustment of the acceleration and braking mechanism of the car, and the technology in this field is relatively mature. Lateral control manipulates the steering wheel of the car by accurately outputting the control quantity, thereby guiding the vehicle to accurately fit the predetermined trajectory, and is committed to achieving the goal of trajectory tracking. Currently popular trajectory tracking control strategies include predictive control, fuzzy control, and adaptive control. Predictive control technology relies on the established prediction model and established constraints to continuously optimize the control instruction sequence through online calculation. However, in the face of computing resource constraints, this technology is often difficult to achieve the ideal calculation speed in actual control. Fuzzy control strategies rely on subjective experience to construct complex fuzzy rules and membership function systems, which makes it difficult for their design to meet the requirements of systematization and reliability. Adaptive control technology also needs to rely on complex dynamic models, which are often difficult to accurately obtain in actual control. With the continuous advancement of technology, researchers have begun to explore more advanced data-driven controller design methods. In recent years, some learning control methods based on neural networks have shown strong performance in control tasks such as robots and autonomous driving. However, due to the uninterpretability of neural networks, directly relying on neural networks for control often lacks guarantees in terms of security and stability, which to a certain extent limits the further application of learning methods.

[0003] As a common controller type in the control field, the PID (Proportional-Integral-Derivative) controller has the characteristics of simple structure, strong robustness and high reliability, and is therefore often used to solve various control problems. Especially in some SISO (Single-Input Single-Output) systems, the PID controller ensures its high computational efficiency and robust performance due to its unique model-free characteristics. However, in some MIMO (Multiple-Input Multiple-Output) tasks, such as vehicle trajectory tracking control, there is often a close coupling relationship between the longitudinal and lateral controls, and it is usually difficult to meet the control requirements of the system by simply adjusting a single parameter. In addition, achieving accurate and reliable tracking control often requires a lot of manual parameter adjustment. Summary of the invention

[0004] In order to solve the technical problems of poor real-time calculation, weak application reliability and reliance on complex dynamic models in current vehicle trajectory tracking control, the embodiment of the present invention provides a PID trajectory tracking control method and device based on reinforcement learning. The technical solution is as follows:

[0005] On the one hand, a PID trajectory tracking control method based on reinforcement learning is provided, the method is implemented by a PID trajectory tracking control device, and the method includes:

[0006] S1. Obtain the current state, reference trajectory and actual operation error of the autonomous driving vehicle to be controlled.

[0007] S2. Construct a strategy model for vehicle trajectory tracking control.

[0008] S3. Input the current state and reference trajectory into the strategy model of vehicle trajectory tracking control to obtain PID control parameters.

[0009] S4. Input the PID control parameters and the actual operation error into the PID controller to obtain the control amount of the autonomous driving vehicle, and complete the trajectory tracking control of the autonomous driving vehicle according to the control amount of the autonomous driving vehicle.

[0010] Optionally, the process of constructing the strategy model of the vehicle trajectory tracking control in S2 includes:

[0011] S201, determine initial conditions and parameter initialization.

[0012] Determining the initial conditions includes: determining the discrete interval and the prediction time domain, and obtaining the reference trajectory, the expected speed information, and the vehicle state.

[0013] Parameter initialization includes: initializing PID controller and reinforcement learning strategy.

[0014] S202, after determining the initial conditions and initializing the parameters, obtain a continuous curve of the reference trajectory in the vehicle coordinate system described by a mathematical equation, determine the target point to be tracked at the current moment according to the continuous curve of the reference trajectory, and obtain the coordinates of the target point to be tracked at the current moment in the geodetic coordinate system.

[0015] S203, determining a prediction reference point according to the coordinates of the target point being tracked at the current moment in the geodetic coordinates, the discrete interval, the prediction time domain, and the preset prediction horizontal coordinate difference, and obtaining reference information of the prediction reference point.

[0016] S204: Obtain the tracking error at the next moment according to the current state of the vehicle and the reference information of the predicted reference point.

[0017] S205. Construct a target reward function based on the tracking error at the next moment.

[0018] S206: Obtain the tracking target point at the next moment according to the output control amount of the PID controller at the current moment.

[0019] S207. Acquire observation information of the vehicle according to the target point tracked at the next moment.

[0020] S208. Construct an input of a strategy model for vehicle trajectory tracking control based on the vehicle's observation information and the target reward function.

[0021] S209: Setting action constraints of the strategy model for vehicle trajectory tracking control.

[0022] S210, according to the input of the strategy model of the vehicle trajectory tracking control and the action constraints, a reinforcement learning algorithm is used to calculate the target reward function, and then obtain the PID control parameters.

[0023] Optionally, obtaining the tracking error at the next moment according to the current state of the vehicle and the reference information of the predicted reference point in S204 includes:

[0024] According to the current state of the vehicle, the current PID controller parameters are obtained through reinforcement learning.

[0025] According to the PID controller parameters at the current moment, the output control quantity of the PID controller at the current moment is obtained.

[0026] According to the current state of the vehicle and the output control amount of the PID controller at the current moment, the state of the vehicle at the next moment is obtained.

[0027] According to the vehicle state at the next moment and the reference information of the predicted reference point, the tracking error at the next moment is obtained.

[0028] Optionally, the target reward function in S205 is as shown in the following formula (1):

[0029] r(X,X ref ,u mv , Δa) = e(X, X ref ) T Pe(X,X ref )+u mv T Qu mv +Δa T RΔa (1)

[0030] In the formula, r(X,X ref ,u mv , Δa) represents the target reward function, X represents the state of the vehicle, X ref Represents the reference information of the prediction reference point, u mv =[ωsteer , a x ] T Represents the output control quantity of the PID controller, ω steer is the steering wheel angle, a x is the output longitudinal acceleration, Δa represents the change in strategy output, e(X, X ref ) represents the tracking error, T represents the matrix transpose, P represents the diagonal positive definite matrix used to control the tracking error, Q represents the diagonal positive definite matrix used to control the output control amount, and R represents the diagonal positive definite matrix used to control the output change of the control strategy.

[0031] Optionally, the tracking error satisfies the following formula (2):

[0032]

[0033] In the formula, y t+1 represents the longitudinal position of the vehicle in the ego-vehicle coordinate system at time t+1, Represents the ordinate in the reference information of the predicted reference point at time t+1, It represents the heading angle of the vehicle in the ego-vehicle coordinate system at time t+1, represents the heading angle in the reference information of the predicted reference point at time t+1, u t+1 represents the longitudinal forward speed of the vehicle in the vehicle coordinate system at time t+1, represents the longitudinal forward speed in the reference information of the predicted reference point at time t+1, Δy t+1 represents the lateral error at time t+1, Indicates the heading angle error at time t+1, Δu t+1 Represents the velocity error at time t+1.

[0034] The change in strategy output satisfies the following formula (3):

[0035]

[0036] In the formula, represents the proportional gain of the steering wheel angle PID controller at time t, represents the integral gain of the steering wheel angle PID controller at time t, represents the differential gain of the steering wheel angle PID controller at time t, represents the proportional gain of the longitudinal acceleration PID controller at time t, represents the integral gain of the longitudinal acceleration PID controller at time t, represents the differential gain of the longitudinal acceleration PID controller at time t, It represents the proportional gain change of the angular velocity PID controller at time t+1, It represents the change of the integral gain of the angular velocity PID controller at time t+1, It represents the differential gain change of the angular velocity PID controller at time t+1, It represents the proportional gain change of the longitudinal acceleration PID controller at time t+1, It represents the change of the integral gain of the longitudinal acceleration PID controller at time t+1, Represents the change in the differential gain of the longitudinal acceleration PID controller at time t+1.

[0037] Optionally, in S210, according to the input of the strategy model of the vehicle trajectory tracking control and the action constraints, a reinforcement learning algorithm is used to calculate the target reward function, and then the PID control parameters are obtained, including:

[0038] According to the input of the strategy model of vehicle trajectory tracking control, action constraints and reinforcement learning algorithm, the Actor-Critic structure is adopted to learn the value distribution objective function and strategy function, calculate the target reward function, and then obtain the PID control parameters.

[0039] Optionally, the PID controller in S4 includes: a lateral error PID controller and a speed error PID controller.

[0040] Among them, the lateral error PID controller is shown in the following formula (4):

[0041]

[0042] In the formula, Δω steer Represents the change in steering wheel angle, Δy t represents the lateral error at time t, represents the ratio of the steering wheel angle PID controller at time t, represents the integral of the steering wheel angle PID controller at time t, Represents the differential gain of the steering wheel angle PID controller at time t.

[0043] The speed error PID controller is shown in equation (5):

[0044]

[0045] In the formula, Δa x Indicates the change in longitudinal acceleration, Δu t represents the longitudinal velocity error at time t, Represents the ratio of the longitudinal acceleration PID controller at time t, represents the integral of the longitudinal acceleration PID controller at time t, Represents the differential gain of the longitudinal acceleration PID controller at time t.

[0046] On the other hand, a PID trajectory tracking control device based on reinforcement learning is provided, and the device is applied to a PID trajectory tracking control method based on reinforcement learning, and the device includes:

[0047] The acquisition module is used to obtain the current state, reference trajectory and actual operation error of the autonomous driving vehicle to be controlled.

[0048] Building module, used to build the strategy model for vehicle trajectory tracking control.

[0049] The input module is used to input the current state and reference trajectory into the strategy model of vehicle trajectory tracking control to obtain PID control parameters.

[0050] The output module is used to input the PID control parameters and the actual operation error into the PID controller to obtain the control quantity of the autonomous driving vehicle, and complete the trajectory tracking control of the autonomous driving vehicle according to the control quantity of the autonomous driving vehicle.

[0051] On the other hand, a PID trajectory tracking control device is provided, which includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned PID trajectory tracking control methods based on reinforcement learning is implemented.

[0052] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned PID trajectory tracking control methods based on reinforcement learning.

[0053] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0054] In the embodiment of the present invention, the coupling of multi-PID control is given by the reinforcement learning algorithm, and the PID application, which is already widely used, is expanded to the related control tasks of MIMO, and the application range is wide; the present invention solves the error amount based on the reference information at the prediction time, eliminates the lag of error feedback, and the control is timely and reliable; the present invention retains the nonlinear characteristics of the controlled object model when constructing the control problem, and the control accuracy is high; the present invention converts the online optimization problem into the offline solution of the policy network parameters by designing the loss function of the offline training policy network, and does not need to store a large amount of state to control amount mapping relationship, thus saving memory resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 It is a flow chart of a PID trajectory tracking control method based on reinforcement learning provided by an embodiment of the present invention;

[0057] Figure 2 It is a schematic diagram of the overall process flow of the MIMO variable parameter PID trajectory tracking control method based on reinforcement learning provided by an embodiment of the present invention;

[0058] Figure 3 It is a schematic diagram of a MIMO variable parameter PID trajectory tracking control method based on reinforcement learning provided by an embodiment of the present invention;

[0059] Figure 4 1 is a schematic diagram (I) showing a comparison of trajectory tracking effects of a MIMO variable parameter PID trajectory tracking control method based on reinforcement learning provided by an embodiment of the present invention;

[0060] Figure 5 1 is a schematic diagram (II) showing a comparison of trajectory tracking effects of a MIMO variable parameter PID trajectory tracking control method based on reinforcement learning provided by an embodiment of the present invention;

[0061] Figure 6 It is a block diagram of a PID trajectory tracking control device based on reinforcement learning provided by an embodiment of the present invention;

[0062] Figure 7 It is a structural schematic diagram of a PID trajectory tracking control device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0064] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0065] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0066] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0067] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0068] The embodiment of the present invention provides a PID trajectory tracking control method based on reinforcement learning, which can be implemented by a PID trajectory tracking control device, which can be a terminal or a server. Figure 1 The flowchart of the PID trajectory tracking control method based on reinforcement learning is shown in the figure. The processing flow of the method may include the following steps:

[0069] First, there are several key terms in reinforcement learning that need to be explained:

[0070] 1. Environment: refers to the external environment that the agent interacts with. It can be a physical environment in the real world, or a virtual environment or a simulated environment. The environment will affect the state and actions of the agent and give corresponding feedback based on the agent's actions.

[0071] 2. Agent: refers to the subject involved in the learning and decision-making process, which can be a robot, a software program, or any entity that can perceive and make decisions. The agent observes the state of the environment and chooses appropriate actions to maximize the reward.

[0072] 3. State: represents the current situation or characteristics of the environment, which can be fully observable or partially observable. The agent selects actions based on the current state, and the environment gets the next state based on the results of the action.

[0073] 4. Observation: This refers to the information about the state that the agent obtains from the environment. In some cases, the state may not be directly observable, and the agent needs to approximate or infer the state through observation.

[0074] 5. Action: refers to the behavior or decision that the agent can perform in a specific state. The goal of the agent to choose an action is to maximize the reward.

[0075] 6. Reward: Represents the immediate feedback that the agent receives after performing a specific action in a specific state. The reward can be positive, negative, or zero and is used to evaluate the quality of the action. The goal of the agent is to maximize the cumulative reward by choosing actions.

[0076] 7. Objective Function: A value function constructed based on rewards, which is used to guide the learning and decision-making process of the agent. The goal of the agent is to improve its strategy by optimizing the value function based on rewards to maximize the cumulative rewards.

[0077] By understanding and applying these terms, reinforcement learning algorithms enable an agent to gradually improve its strategy in its interactions with the environment in order to maximize the total reward.

[0078] In the reinforcement learning of the present invention, the environment can be a system that simulates the real situation by performing state deduction on the objects therein based on laws such as physical laws, kinematics and geometric constraints. This system consists of agents, obstacles, map constraints, etc. Agents are object elements in the environment that can act autonomously, accept observations and give actions, such as autonomous driving vehicles. States are basic attributes of elements in the environment, such as the position, orientation and speed of vehicles. Observations are the encoding features of the states of other elements in the environment from the perspective of the agent based on a given state. Actions are the output of the agent, which is determined by the agent's strategy. Deduction refers to the application of dynamic equations in each time step to iteratively calculate the state of each step given an action sequence and the initial state of the object, and finally obtain a state sequence as output. The state transition model is defined on the object, and given the state and action of the object at the current moment, the state to which the object will transfer at the next moment is output. The model is usually restricted by laws such as kinematic constraints. The reward represents the evaluation of the state and action of the elements in the environment at the current time step, and is a scalar value. The cumulative reward is the sum calculated by accumulating the rewards of each time step starting from the current state until the end of the deduction.

[0079] The vehicle trajectory tracking optimal control model is constructed under the reinforcement learning framework, aiming to learn an optimal control strategy by allowing the vehicle to interact with the environment, so as to maximize the objective function obtained by executing the control strategy from the initial moment. The vehicle trajectory tracking control model includes a policy network, a PID controller, and a state transition model. The policy network outputs the optimal PID control parameters under the current state according to the input vehicle state and the reference trajectory; the PID controller outputs the vehicle control quantity according to the current error information and the optimal parameters of the policy network; the state transition model outputs the vehicle state at the next moment given the vehicle state at the current moment and the control quantity of the PID controller, which is determined by formulas such as the state space equation.

[0080] S1. Obtain the current state, reference trajectory and actual operation error of the autonomous driving vehicle to be controlled.

[0081] S2. Constructing a strategy model for vehicle trajectory tracking control. Optionally, the process of constructing the strategy model for vehicle trajectory tracking control in S2 may include S201-S210:

[0082] S201, determine initial conditions and parameter initialization.

[0083] In a feasible implementation manner, the above step S201 is initialization of the environment, state and various parameters.

[0084] Specifically, at time t, the discrete interval and prediction time domain N of the vehicle trajectory tracking control system are determined, and the continuous reference trajectory, expected speed information and vehicle position state in the existing geodetic coordinate system are input into the vehicle trajectory tracking control system.

[0085] The reference trajectory information can be given by the trajectory planning module; the expected speed information can be given by the human-computer interaction module or the speed planning module; the vehicle position state can be given by the odometer and other state estimation and positioning modules, such as the initial state of the vehicle X0 = [0, 0, 0, 0, 0] T .

[0086] Furthermore, the internal PID controller and reinforcement learning strategy are initialized.

[0087] Specifically, the PID controller sets the initial six-dimensional parameters for angular velocity and acceleration control: At the same time, the reinforcement learning observations are initialized and a 3N+12-dimensional vector is created for subsequent data storage. Similarly, a 6-dimensional vector is created for the policy output to update the parameters of the PID controller in real time.

[0088] S202, after determining the initial conditions and initializing the parameters, obtain a continuous curve of the reference trajectory in the vehicle coordinate system described by a mathematical equation, determine the target point to be tracked at the current moment according to the continuous curve of the reference trajectory, and obtain the coordinates of the target point to be tracked at the current moment in the geodetic coordinate system.

[0089] In a feasible implementation manner, the above step S202 is to obtain the tracking target point at the current moment.

[0090] Specifically, at time t, after initialization, the continuous curve described by the mathematical equation of the reference trajectory point in the vehicle coordinate system is obtained, and the closest intersection point of the continuous curve with the reference trajectory point in the y-axis direction of the vehicle coordinate system is taken as the current tracking target point, and the horizontal coordinate of the current tracking target point is converted into the horizontal coordinate of the current tracking target point in the geodetic coordinate by the following formula (1) (2) through the current vehicle state:

[0091]

[0092] In the formula, x0 and y0 represent the horizontal and vertical coordinates of the target point being tracked at the current moment in geodetic coordinates. represents the horizontal and vertical coordinates of the target point in the vehicle coordinate system at time t (according to the above tracking target point selection principle, ), Indicates the yaw angle measured at the current moment.

[0093] S203, determining a prediction reference point according to the coordinates of the target point being tracked at the current moment in the geodetic coordinates, the discrete interval, the prediction time domain, and the preset prediction horizontal coordinate difference, and obtaining reference information of the prediction reference point.

[0094] In a feasible implementation manner, the above step S203 is to calculate a reference quantity in the prediction time domain.

[0095] Specifically, in the geodetic coordinate system, the manually set discrete intervals and the preview horizontal coordinate differences manually set according to the expected preview effects are obtained, and x0 is used as the starting point. The horizontal coordinates of each preview reference point are obtained through the discrete intervals and the preview horizontal coordinate differences, and the various reference quantities such as the preview reference point vertical coordinates and yaw angles are calculated and stored in a queue.

[0096] For the reference trajectory of step S202, the horizontal coordinate of the preview reference point is obtained by the following formula:

[0097]

[0098] In the formula, It represents the horizontal coordinate of the kth preview reference point (prediction reference point), N is the prediction time domain, dx is the discrete interval (prediction horizontal coordinate interval), and x0 represents the horizontal coordinate of the current vehicle in the geodetic coordinate system.

[0099] Furthermore, according to the prediction time domain, we can obtain Compare x0 with Substitution Other reference quantities on the reference trajectory, such as the ordinate and heading angle, can be obtained.

[0100] For example, suppose that at t = 0, the current vehicle horizontal coordinate x0 = 0, the predicted horizontal coordinate interval is 0.5m, and the reference trajectory is y ref = sin(πx ref / 25), the prediction time domain is N=10; the horizontal coordinate in the reference information of the first prediction reference point is: The vertical axis is: The heading angle is:

[0101] S204: Obtain the tracking error at the next moment according to the current state of the vehicle and the reference information of the predicted reference point.

[0102] Optionally, the above step S204 may include the following steps S2041-S2044:

[0103] S2041. According to the current state of the vehicle, the current PID controller parameters are obtained through reinforcement learning.

[0104] S2042. Obtain the output control amount of the PID controller at the current moment according to the PID controller parameters at the current moment.

[0105] S2043. Obtain the state of the vehicle at the next moment according to the state of the vehicle at the current moment and the output control amount of the PID controller at the current moment.

[0106] S2044: Obtain the tracking error at the next moment according to the vehicle state at the next moment and the reference information of the predicted reference point.

[0107] In a feasible implementation manner, the above step S204 is to obtain the vehicle state in discrete time.

[0108] Specifically, the vehicle coordinate system is established with the vehicle position at time t as the origin, and the vehicle state at time t+1 is obtained based on the vehicle dynamics equation and the output action of the trajectory tracking controller.

[0109] Among them, the vehicle state X at time t t Contains the horizontal and vertical position x of the vehicle in the vehicle coordinate system at time t t ,y t , heading angle Longitudinal forward speed u t , lateral sideslip speed v t and angular velocity ωt ,Right now

[0110] Furthermore, the strategy output at time t includes the following: t Feedback and speed error Δu t Feedback PID controller parameters Make the PID controller output the relevant control quantity u mv =[ω steer , a x ] T To achieve coupling control; where ω steer is the steering wheel angle, and ax is the output longitudinal acceleration.

[0111] S205. Construct a target reward function based on the tracking error at the next moment.

[0112] In a feasible implementation manner, the above step S205 is to construct a target reward function.

[0113] Specifically, the target reward function is constructed using the tracking error at time t+1 obtained in step S204.

[0114] Among them, the reward function r(X, X ref ,u mv , Δa) is designed as the tracking error e(X, X ref ), control amount u mv , the quadratic weighted sum of the strategy output change Δa, that is:

[0115] r(X,X ref ,u mv , Δa) = e(X, X ref ) T Pe(X,X ref )+u mv T Qu mv +Δa T RΔa (4)

[0116] In the formula, r(X, X ref ,u mv , Δa) represents the target reward function, X represents the state of the vehicle, X ref Represents the reference information of the prediction reference point, u mv =[ω steer , a x ] T Represents the output control quantity of the PID controller, ω steer is the steering wheel angle, a x is the output longitudinal acceleration, Δa represents the change in strategy output, e(X, X ref) represents the tracking error, T represents the matrix transpose, P, Q, and R are all diagonal positive definite matrices, which are used to control the weight of the tracking error and the influence of the control amount on the control result.

[0117] Among them, the state tracking error satisfies the following formula (5):

[0118]

[0119] In the formula, y t+1 represents the longitudinal position of the vehicle in the ego-vehicle coordinate system at time t+1, Represents the ordinate in the reference information of the predicted reference point at time t+1, It represents the heading angle of the vehicle in the ego-vehicle coordinate system at time t+1, represents the heading angle in the reference information of the predicted reference point at time t+1, u t+1 represents the longitudinal forward speed of the vehicle in the vehicle coordinate system at time t+1, represents the longitudinal forward speed in the reference information of the predicted reference point at time t+1, Δy t+1 represents the lateral error at time t+1, Indicates the heading angle error at time t+1, Δu t+1 Represents the velocity error at time t+1.

[0120] For the action fluctuation, the following formula (6) is satisfied:

[0121]

[0122] In the formula, represents the proportional gain of the steering wheel angle PID controller at time t, represents the integral gain of the steering wheel angle PID controller at time t, represents the differential gain of the steering wheel angle PID controller at time t, represents the proportional gain of the longitudinal acceleration PID controller at time t, represents the integral gain of the longitudinal acceleration PID controller at time t, represents the differential gain of the longitudinal acceleration PID controller at time t, It represents the proportional gain change of the angular velocity PID controller at time t+1, It represents the change of the integral gain of the angular velocity PID controller at time t+1, It represents the differential gain change of the angular velocity PID controller at time t+1, It represents the proportional gain change of the longitudinal acceleration PID controller at time t+1, It represents the change of the integral gain of the longitudinal acceleration PID controller at time t+1, Represents the change in the differential gain of the longitudinal acceleration PID controller at time t+1.

[0123] S206: Obtain the tracking target point at the next moment according to the output control amount of the PID controller at the current moment.

[0124] In a feasible implementation manner, the above step S206 is updating the reference trajectory points.

[0125] Specifically, based on the vehicle dynamics equation, according to the PID controller output u at the previous moment mv The calculated horizontal coordinate x of the vehicle at time t+1 0,t+1 , obtain a new reference point information sequence.

[0126] S207. Acquire observation information of the vehicle according to the target point tracked at the next moment.

[0127] In a feasible implementation manner, the above step S207 is to obtain control model observations.

[0128] Specifically, the vehicle's observations consist of the following four parts: the vehicle's own observations X t , predicted trajectory reference information X ref , error tracking observation E t and the strategy output a at the previous moment t-1 .

[0129] Among them, the ego-vehicle observation X t The information can be obtained in step S203, and the predicted trajectory reference information X ref It can be obtained in step S205, including reference information of N moments k∈1,2...N; error tracking observation

[0130] E t =[Δy t -Δy t-1 , Δy t , Δy t -2Δy t-1 +Δy t-2 , Δu t -Δu t-1 , Δu t , Δu t -2Δu t-1 +Δu t-2 ] T And the strategy output at the previous moment It can be obtained by step S205.

[0131] S208. Construct an input of a strategy model for vehicle trajectory tracking control based on the vehicle's observation information and the target reward function.

[0132] In a feasible implementation, the observation information obtained in step S207 and the reward function in step S205 are used as inputs of the vehicle trajectory tracking control model, and the observation information is input into the control model.

[0133] S209: Setting action constraints of the strategy model for vehicle trajectory tracking control.

[0134] In a feasible implementation, in order to ensure the stability of vehicle control, the action is set to a bounded output, and finally the vehicle state space equation with control quantity constraints can be obtained:

[0135]

[0136] Among them, A is the state matrix, B is the input matrix, X k represents the state of the vehicle at time k, and a and b are the lower and upper bounds of the control quantity.

[0137] S210, according to the input of the strategy model of vehicle trajectory tracking control, action constraints and reinforcement learning algorithm, adopt the Actor-Critic structure, learn the value distribution objective function and strategy function, calculate the target reward function, and then obtain the PID control parameters.

[0138] The value distribution reinforcement learning algorithm uses the Actor-Critic structure to learn independent value distribution functions and policy functions, where the mathematical expression of the value distribution objective function is as follows:

[0139]

[0140] u=π θ (X,X ref ;ω) (9)

[0141] Z(X,X ref ,u)=r+γG t+1 (10)

[0142]

[0143] In formula (8), Qz is the target value function of vehicle trajectory tracking control, X, X ref and u are the ego vehicle state, reference trajectory and ego vehicle action at any time in the infinite time domain, θ is the parameter to be optimized in the parameterized target value network, and B is the experience pool used to store the training samples (X, X) generated by the interaction between the strategy and the environment. ref ,u,r,X′,X′ ref ), r is the reward function, is the Bellman self-consistent operator, and θ′ represents the parameters of the old value network before updating.

[0144] In formula (9), π θ (X,X ref ω) is a parameterized policy network, that is, the input of the vehicle state X and the environment state X at any time ref The action u of the vehicle can be obtained, ω is the parameter to be optimized in the parameterized strategy, and u′ represents the action of the vehicle by another strategy network π ω′ Output action.

[0145] In formula (10), formula (11), and formula (12), Z is the cumulative return generated based on the flexible strategy π, which is the state-action return, denoted as Zσ(X, X ref ,u),γ is the discount factor; is the probability density function of the flexible Z value, G t represents the random flexible return of strategy π, α represents the strategy entropy temperature coefficient, σ Z represents the variance of the cumulative return Z.

[0146] Strategy function, where the mathematical expression of the strategy objective function is as follows:

[0147]

[0148] Among them, Q is given by the value distribution function as the policy entropy temperature coefficient, and its update rule is as follows:

[0149]

[0150] Among them, β c is the learning rate, is the given policy entropy target value.

[0151] like Figure 2 , Figure 3 As shown, in the specific implementation process, the reinforcement learning part of the vehicle's trajectory tracking control needs to be solved iteratively, which specifically includes the following steps:

[0152] Step 10-1: Give relevant initial conditions and parameter initialization.

[0153] The initial state of the vehicle can be given directly, such as X0 = [0, 0, 0, 0] T , can also be randomly generated within a reasonable range; given the reference trajectory y ref = sin(πx ref \25), refer to the expected speed u ref =5m / s, prediction interval dx=0.5m, prediction time domain N=10; initialization time t, given experience pool B size is 10 6 And randomly initialized to store the training samples (X, X) generated by the interaction between the strategy and the environment ref,u,r,X′,X′ ref ); Initialize the value distribution network parameter θ, the policy network parameter ω and the policy entropy coefficient α; Initialize the target network parameters θ′←θ, ω′←ω; Give a suitable learning rate β a =10 -5 , β b =10 -5 , τ=10 -5 And error ∈ = 10 -5 ; Initialize the iteration number k = 0. The value distribution network and the policy network both use fully connected multi-layer perceptrons with a hidden layer size of 256×256.

[0154] Step 10-2: Obtain reference quantities in the prediction time domain.

[0155] The N reference trajectory points in the geodetic coordinate system in the prediction time domain are obtained by using the horizontal coordinate interval length dx and the prediction step length N in discrete time, which are

[0156] Step 10-3: Sampling.

[0157] Loop index variable k = k + 1, in the current state X t Using strategy π ω Interact with the environment to obtain a reward function and observe the next state X t+1 , and obtain a new reference point X t ,u t , R t+1 , X t+1 , Form an experience sample and store the samples into experience pool B one by one.

[0158] Step 10-4, solve the value distribution objective function value.

[0159] The multiple experience samples sampled from the experience replay pool B are used as the initial values ​​of the forward solution process. The value distribution network shown in formula (8) is used to calculate the objective function values ​​respectively, providing initial values ​​for the backward solution value distribution network gradient and update parameters.

[0160] Step 10-5: Update the value distribution network parameters.

[0161] The objective function value obtained is used to calculate the gradient. After averaging the gradient of the batch-calculated value network, the network parameters are gradient-decreased. The update rule of the value network in the kth iteration is:

[0162]

[0163] In the formula, represents the θ gradient operator, θk Represents the objective function value in the kth iteration.

[0164] Step 10-6: Update policy network parameters.

[0165] If k is a positive integer multiple of m, then use equation (15) to batch calculate the objective function and solve the policy gradient, and use the averaged policy gradient to perform gradient descent on the policy parameters. The update rule of the policy network in the Kth iteration is:

[0166]

[0167] In the formula, β b Indicates the learning rate, represents the ω gradient operator.

[0168] Step 10-8, update the strategy entropy coefficient.

[0169] If k is a positive integer multiple of m, then update is performed using equation (14).

[0170] Step 10-9: Update the target value network and strategy network parameters.

[0171] If k is a positive integer multiple of m, then update according to the following rules:

[0172] θ′ k+1 =-(1-τ)θ k ′+τθ k (18)

[0173] ω′ k+1 =-(1-τ)ω k ′+τω k (19)

[0174] Step 10-10: Determine whether the result converges.

[0175] If |J π (ω k+1 )-J π (ω k )|>∈, it indicates that the strategy has not converged, and return to step 10-3 to continue solving; otherwise, it indicates that the strategy has converged within the error range, and the output is the optimal parameterized strategy.

[0176] S211. Output network results.

[0177] The parameterized network is used as the output of the policy network. The policy network can output the tuning parameters of the vehicle trajectory tracking coupled PID control model according to the vehicle state and prediction information.

[0178] S3. Input the current state and reference trajectory into the strategy model of vehicle trajectory tracking control to obtain PID control parameters.

[0179] S4. Input the PID control parameters and the actual operation error into the PID controller to obtain the control amount of the autonomous driving vehicle, and complete the trajectory tracking control of the autonomous driving vehicle according to the control amount of the autonomous driving vehicle.

[0180] In a feasible implementation manner, the output of step S210 is Input into the two coupled PID controllers of lateral error and speed error feedback to realize MIMO coupled control. The mathematical expressions of lateral error and speed error PID controllers are as follows:

[0181]

[0182] It is worth noting that equations (20) and (21) provide feedback based on the predicted error at time t+1 obtained in step S203, eliminating the lag of error feedback and improving control accuracy and effect.

[0183] The present invention adopts a reinforcement learning algorithm and utilizes a parameterized strategy network obtained through offline training to directly output the optimal PID parameters of the vehicle according to the current state of the vehicle and predicted reference information, so that the PID control output under the coupling parameters realizes the change of the vehicle state space, and continuously outputs the control amount according to the vehicle state and environmental information, thereby realizing high real-time and high-precision trajectory tracking control of the vehicle.

[0184] The present invention uses the optimal parameterized strategy network learned after 400,000 iterations of the vehicle trajectory tracking control model to simulate and control the vehicle at u ref =5m / s expected longitudinal speed to track the given reference curve, and compare the results obtained with the results of uncoupled PID control under the same reference trajectory and reference speed, such as Figure 4 , Figure 5 As shown in the figure, the results show that the method proposed in the present invention achieves good tracking effects on both position and speed in the entire trajectory tracking process; compared with the PID control under non-coupling, the ordinary PID exposes the problem of control lag in the control, resulting in poor tracking effect at the sudden change of trajectory corners, and in speed control, due to the coupling problem, weak oscillation is re-generated after convergence; however, the method of the present invention solves the above problems better due to the introduction of preview error and reinforcement learning.

[0185] The present invention proposes a construction and solution method for variable parameter PID trajectory tracking control based on reinforcement learning. First, the entire control system identifies the closest point between the vehicle's lateral position and the reference trajectory under the assumption of a given expected speed and reference trajectory, and then determines the vehicle's current and expected optimal position state; then, the PID coupling parameters are output through the reinforcement learning algorithm; the obtained PID parameters are input into the PID controller, and the vehicle's next state is predicted according to the discrete dynamic characteristics of the vehicle, so as to realize vehicle trajectory tracking control.

[0186] The core of the present invention is to use the current state of the vehicle, the expected trajectory information, and the lateral deviation set at different time points as input data, and the network strategy learned offline through reinforcement learning, and the various PID parameters are dynamically adjusted, and finally the control quantity is output by each PID controller. This step converts the complex online optimization problem of the common trajectory tracking algorithm into a strategy network parameter problem that can be solved offline through reinforcement learning. When applied online, the on-board control system can directly output and adjust the optimal PID control parameters through the strategy network in real time according to the current state of the vehicle and the actual operating error, and then achieve high robustness control of the vehicle behavior through PID control. This process ensures that the PID controller can achieve high-precision control effects of multi-variable coupling and realize efficient online real-time trajectory tracking optimization while ensuring stability and reliability.

[0187] The present invention has the advantages of high computational efficiency (the strategy function only needs to approximately fit the optimal parameters), strong reliability (the PID controller itself has good anti-interference ability), high precision (the strategy learning process retains the nonlinear characteristics of the controlled object model), and wide application range (the method takes the simultaneous elimination of multiple coupling errors as the control goal and is not limited to trajectory tracking tasks). It breaks through the complexity of implementing PID to achieve MIMO coupling control and the problems of poor efficiency in online solution of existing trajectory tracking control problems, complex design and implementation rules, and reliability that needs to be strictly proven.

[0188] In the embodiment of the present invention, the coupling of multi-PID control is given by the reinforcement learning algorithm, and the PID application, which is already widely used, is expanded to the related control tasks of MIMO, and the application range is wide; the present invention solves the error amount based on the reference information at the prediction time, eliminates the lag of error feedback, and the control is timely and reliable; the present invention retains the nonlinear characteristics of the controlled object model when constructing the control problem, and the control accuracy is high; the present invention converts the online optimization problem into the offline solution of the policy network parameters by designing the loss function of the offline training policy network, and does not need to store a large amount of state to control amount mapping relationship, thus saving memory resources.

[0189] Figure 61 is a block diagram of a PID trajectory tracking control device based on reinforcement learning according to an exemplary embodiment, and the device is used in a PID trajectory tracking control method based on reinforcement learning. Figure 6 The device includes an acquisition module 310, a construction module 320, an input module 330 and an output module 340.

[0190] in:

[0191] The acquisition module 310 is used to obtain the current state, reference trajectory and actual operation error of the autonomous driving vehicle to be controlled.

[0192] The construction module 320 is used to construct a strategy model for vehicle trajectory tracking control.

[0193] The input module 330 is used to input the current state and the reference trajectory into the strategy model of the vehicle trajectory tracking control to obtain PID control parameters.

[0194] The output module 340 is used to input the PID control parameters and the actual operation error into the PID controller to obtain the control amount of the autonomous driving vehicle, and complete the trajectory tracking control of the autonomous driving vehicle according to the control amount of the autonomous driving vehicle.

[0195] In the embodiment of the present invention, the coupling of multi-PID control is given by the reinforcement learning algorithm, and the PID application, which is already widely used, is expanded to the related control tasks of MIMO, and the application range is wide; the present invention solves the error amount based on the reference information at the prediction time, eliminates the lag of error feedback, and the control is timely and reliable; the present invention retains the nonlinear characteristics of the controlled object model when constructing the control problem, and the control accuracy is high; the present invention converts the online optimization problem into the offline solution of the policy network parameters by designing the loss function of the offline training policy network, and does not need to store a large amount of state to control amount mapping relationship, thus saving memory resources.

[0196] Figure 7 is a schematic diagram of the structure of a PID trajectory tracking control device provided by an embodiment of the present invention, such as Figure 7 As shown, the PID trajectory tracking control device may include the above Figure 6 The PID trajectory tracking control device based on reinforcement learning is shown. Optionally, the PID trajectory tracking control device 410 may include a first processor 2001.

[0197] Optionally, the PID trajectory tracking control device 410 may further include a memory 2002 and a transceiver 2003 .

[0198] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0199] Combine the following Figure 7 The various components of the PID trajectory tracking control device 410 are specifically introduced:

[0200] The first processor 2001 is the control center of the PID trajectory tracking control device 410, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or can be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0201] Optionally, the first processor 2001 may perform various functions of the PID trajectory tracking control device 410 by running or executing a software program stored in the memory 2002 , and calling data stored in the memory 2002 .

[0202] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 7 CPU0 and CPU1 are shown in FIG.

[0203] In a specific implementation, as an embodiment, the PID trajectory tracking control device 410 may also include multiple processors, such as Figure 7 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0204] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0205] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be controlled by the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0206] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0207] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 7 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0208] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and track the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0209] It should be noted that Figure 7 The structure of the PID trajectory tracking control device 410 shown in the figure does not constitute a limitation on the router, and the actual knowledge structure identification device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0210] In addition, the technical effects of the PID trajectory tracking control device 410 can refer to the technical effects of the PID trajectory tracking control method based on reinforcement learning described in the above method embodiment, which will not be repeated here.

[0211] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0212] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0213] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0214] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0215] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0216] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0217] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0218] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0219] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0220] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0221] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0222] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0223] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A PID trajectory tracking control method based on reinforcement learning, characterized in that: The method comprises: S1. Obtain the current state, reference trajectory and actual operation error of the autonomous driving vehicle to be controlled; S2. Construct a strategy model for vehicle trajectory tracking control; S3, inputting the current state and reference trajectory into a strategy model of vehicle trajectory tracking control to obtain PID control parameters; S4. Input the PID control parameters and the actual operation error into the PID controller to obtain the control amount of the autonomous driving vehicle, and complete the trajectory tracking control of the autonomous driving vehicle according to the control amount of the autonomous driving vehicle.

2. The PID trajectory tracking control method based on reinforcement learning according to claim 1 is characterized in that: The process of constructing the strategy model of the vehicle trajectory tracking control in S2 includes: S201, determining initial conditions and parameter initialization; Wherein, the determining of the initial conditions includes: determining the discrete interval and the prediction time domain, obtaining the reference trajectory, the expected speed information and the vehicle state; The parameter initialization includes: initializing a PID controller and a reinforcement learning strategy; S202, after determining the initial conditions and initializing the parameters, obtaining a continuous curve of a reference trajectory in a vehicle coordinate system described by a mathematical equation, determining a target point to be tracked at a current moment according to the continuous curve of the reference trajectory, and obtaining the coordinates of the target point to be tracked at a current moment in geodetic coordinates; S203, determining a prediction reference point according to the coordinates of the tracking target point at the current moment in the geodetic coordinates, the discrete interval, the prediction time domain, and the preset prediction horizontal coordinate difference, and obtaining reference information of the prediction reference point; S204, obtaining a tracking error at the next moment according to the current state of the vehicle and the reference information of the predicted reference point; S205, constructing a target reward function according to the tracking error at the next moment; S206, obtaining the tracking target point at the next moment according to the output control amount of the PID controller at the current moment; S207, obtaining observation information of the vehicle according to the next moment tracking target point; S208, constructing an input of a strategy model for vehicle trajectory tracking control according to the observation information of the vehicle and the target reward function; S209, setting action constraints of the strategy model for vehicle trajectory tracking control; S210: According to the input of the strategy model of the vehicle trajectory tracking control and the action constraints, a reinforcement learning algorithm is used to calculate the target reward function, and then obtain PID control parameters.

3. The PID trajectory tracking control method based on reinforcement learning according to claim 2 is characterized in that: The step S204 of obtaining the tracking error at the next moment according to the current state of the vehicle and the reference information of the predicted reference point includes: According to the current state of the vehicle, the current PID controller parameters are obtained through reinforcement learning; According to the current PID controller parameters, the output control amount of the current PID controller is obtained; According to the current state of the vehicle and the output control amount of the PID controller at the current moment, the state of the vehicle at the next moment is obtained; According to the vehicle state at the next moment and the reference information of the predicted reference point, the tracking error at the next moment is obtained.

4. The PID trajectory tracking control method based on reinforcement learning according to claim 2 is characterized in that: The target reward function in S205 is shown in the following formula (1): r(X,X ref ,u mv ,Δa)=e(X,X ref ) T For(X,X ref )+u mv T Qu mv +Δa T RΔa (1) In the formula, r(X,X ref ,u mv , Δa) represents the target reward function, X represents the state of the vehicle, X ref Represents the reference information of the prediction reference point, u mv =[ω steer , a x ] T Represents the output control quantity of the PID controller, ω steer is the steering wheel angle, a x is the output longitudinal acceleration, Δa represents the change in strategy output, e(X, X ref ) represents the tracking error, T represents the matrix transpose, P represents the diagonal positive definite matrix used to control the tracking error, Q represents the diagonal positive definite matrix used to control the output control amount, and R represents the diagonal positive definite matrix used to control the output change of the control strategy.

5. The PID trajectory tracking control method based on reinforcement learning according to claim 4 is characterized in that: The tracking error satisfies the following formula (2): In the formula, y t+1 represents the longitudinal position of the vehicle in the ego-vehicle coordinate system at time t+1, Represents the ordinate in the reference information of the predicted reference point at time t+1, It represents the heading angle of the vehicle in the ego-vehicle coordinate system at time t+1, represents the heading angle in the reference information of the predicted reference point at time t+1, u t+1 represents the longitudinal forward speed of the vehicle in the vehicle coordinate system at time t+1, represents the longitudinal forward speed in the reference information of the predicted reference point at time t+1, Δy t+1 represents the lateral error at time t+1, Indicates the heading angle error at time t+1, Δu t+1 Represents the speed error at time t+1; The strategy outputs a change amount that satisfies the following formula (3): In the formula, represents the proportional gain of the steering wheel angle PID controller at time t, represents the integral gain of the steering wheel angle PID controller at time t, represents the differential gain of the steering wheel angle PID controller at time t, represents the proportional gain of the longitudinal acceleration PID controller at time t, represents the integral gain of the longitudinal acceleration PID controller at time t, represents the differential gain of the longitudinal acceleration PID controller at time t, It represents the proportional gain change of the angular velocity PID controller at time t+1, It represents the integral gain change of the angular velocity PID controller at time t+1, It represents the differential gain change of the angular velocity PID controller at time t+1, It represents the proportional gain change of the longitudinal acceleration PID controller at time t+1, It represents the change of the integral gain of the longitudinal acceleration PID controller at time t+1, Represents the change in the differential gain of the longitudinal acceleration PID controller at time t+1.

6. The PID trajectory tracking control method based on reinforcement learning according to claim 2 is characterized in that: The step S210 uses a reinforcement learning algorithm to calculate the target reward function based on the input of the vehicle trajectory tracking control strategy model and the action constraints, thereby obtaining PID control parameters, including: According to the input of the strategy model of the vehicle trajectory tracking control, action constraints and reinforcement learning algorithm, an Actor-Critic structure is adopted to learn the value distribution objective function and the strategy function, calculate the target reward function, and then obtain the PID control parameters.

7. The PID trajectory tracking control method based on reinforcement learning according to claim 1 is characterized in that: The PID controller in S4 includes: a lateral error PID controller and a speed error PID controller; The lateral error PID controller is as shown in the following formula (4): In the formula, Δω steer Represents the change in steering wheel angle, Δy t represents the lateral error at time t, represents the ratio of the steering wheel angle PID controller at time t, represents the integral of the steering wheel angle PID controller at time t, represents the differential gain of the steering wheel angle PID controller at time t; The speed error PID controller is shown in the following formula (5): In the formula, Δa x Indicates the change in longitudinal acceleration, Δu t represents the longitudinal velocity error at time t, Represents the ratio of the longitudinal acceleration PID controller at time t, represents the integral of the longitudinal acceleration PID controller at time t, Represents the differential gain of the longitudinal acceleration PID controller at time t.

8. A PID trajectory tracking control device based on reinforcement learning, wherein the PID trajectory tracking control device based on reinforcement learning is used to implement the PID trajectory tracking control method based on reinforcement learning as claimed in any one of claims 1 to 7, characterized in that: The device comprises: An acquisition module, used to obtain the current state, reference trajectory and actual operation error of the autonomous driving vehicle to be controlled; A building module for building a strategy model for vehicle trajectory tracking control; An input module, used for inputting the current state and reference trajectory into a strategy model of vehicle trajectory tracking control to obtain PID control parameters; The output module is used to input the PID control parameters and the actual operation error into the PID controller to obtain the control amount of the autonomous driving vehicle, and complete the trajectory tracking control of the autonomous driving vehicle according to the control amount of the autonomous driving vehicle.

9. A PID trajectory tracking control device, characterized in that: The PID trajectory tracking control device comprises: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images