A PID trajectory tracking control method and device based on reinforcement learning

Abstract

The present invention relates to the field of autonomous driving, and in particular to a PID trajectory tracking control method and device based on reinforcement learning. The method comprises: obtaining the current state, reference trajectory, and actual operating error of an autonomous driving vehicle to be controlled; constructing a strategy model for vehicle trajectory tracking control; inputting the current state and reference trajectory into the strategy model for vehicle trajectory tracking control to obtain PID control parameters; inputting the PID control parameters and actual operating error into a PID controller to obtain the control amount of the autonomous driving vehicle, and completing the trajectory tracking control of the autonomous driving vehicle based on the control amount of the autonomous driving vehicle. The present invention utilizes the powerful learning ability of reinforcement learning combined with the stability and strong robustness of PID itself to achieve high real-time, high stability, and high-precision online calculation of vehicle trajectory tracking control. The method solves the problems of poor real-time calculation, weak application reliability, and reliance on complex dynamic models in current vehicle trajectory tracking control.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a PID trajectory tracking control method and device based on reinforcement learning. Background Art

[0002] Motion control is a core technology in the field of autonomous driving, primarily divided into longitudinal control and lateral control. Longitudinal control involves fine-tuning the vehicle's acceleration and braking mechanisms, and the technology in this area is relatively mature. Lateral control involves precisely outputting control variables to manipulate the vehicle's steering wheel, thereby guiding the vehicle to accurately adhere to a predetermined trajectory, aiming to achieve trajectory tracking. Currently, popular trajectory tracking control strategies include predictive control, fuzzy control, and adaptive control. Predictive control relies on established prediction models and pre-defined constraints, continuously optimizing control instruction sequences through online computation. However, due to computing resource constraints, this technology often struggles to achieve the desired computational speed in practical control. Fuzzy control strategies rely on subjective experience to construct complex fuzzy rules and membership function systems, making their design difficult to achieve systematic and reliable requirements. Adaptive control also relies on complex dynamic models, which are often difficult to accurately obtain in practical control. With the continuous advancement of technology, researchers have begun exploring more advanced data-driven controller design methods. In recent years, some learning control methods based on neural networks have demonstrated strong performance in control tasks such as robotics and autonomous driving. However, due to the unexplainability of neural networks, directly relying on neural networks for control often lacks guarantees in terms of security and stability, which to a certain extent limits the further application of learning methods.

[0003] As a common controller type in the control field, the PID (Proportional-Integral-Derivative) controller is characterized by its simple structure, strong robustness, and high reliability, and is therefore often used to solve various control problems. In particular, in some SISO (Single-Input Single-Output) systems, the PID controller ensures high computational efficiency and robust performance due to its unique model-free nature. However, in some MIMO (Multiple-Input Multiple-Output) tasks, such as vehicle trajectory tracking control, there is often a close coupling relationship between longitudinal and lateral control, and it is usually difficult to meet the control requirements of the system with only a single parameter adjustment. In addition, achieving accurate and reliable tracking control often requires a lot of manual parameter adjustment. Summary of the Invention

[0004] To address the current technical issues of vehicle trajectory tracking control, such as poor real-time computation, weak application reliability, and reliance on complex dynamic models, the present invention provides a PID trajectory tracking control method and device based on reinforcement learning. The technical solution is as follows:

[0005] On the one hand, a PID trajectory tracking control method based on reinforcement learning is provided, which is implemented by a PID trajectory tracking control device and includes:

[0006] S1. Obtain the current state, reference trajectory, and actual operating error of the autonomous driving vehicle to be controlled.

[0007] S2. Construct a strategy model for vehicle trajectory tracking control.

[0008] S3. Input the current state and reference trajectory into the vehicle trajectory tracking control strategy model to obtain PID control parameters.

[0009] S4. Input the PID control parameters and the actual operation error into the PID controller to obtain the control quantity of the autonomous driving vehicle. According to the control quantity of the autonomous driving vehicle, the trajectory tracking control of the autonomous driving vehicle is completed.

[0010] Optionally, the process of constructing the vehicle trajectory tracking control strategy model in S2 includes:

[0011] S201: Determine initial conditions and parameter initialization.

[0012] Determining the initial conditions includes determining the discrete interval and prediction time domain, and obtaining the reference trajectory, expected speed information, and vehicle status.

[0013] Parameter initialization includes: initializing the PID controller and reinforcement learning strategy.

[0014] S202. After determining the initial conditions and initializing the parameters, obtain a continuous curve of the reference trajectory in the vehicle coordinate system described by a mathematical equation, determine the current tracking target point based on the continuous curve of the reference trajectory, and obtain the coordinates of the current tracking target point in the geodetic coordinate system.

[0015] S203: Determine a prediction reference point based on the coordinates of the current tracking target point in geodetic coordinates, the discrete interval, the prediction time domain, and the preset prediction horizontal coordinate difference, and obtain reference information of the prediction reference point.

[0016] S204: Obtain the tracking error at the next moment based on the current vehicle state and the reference information of the predicted reference point.

[0017] S205. Construct a target reward function based on the tracking error at the next moment.

[0018] S206: Obtain the tracking target point at the next moment according to the output control quantity of the PID controller at the current moment.

[0019] S207: Acquire the observation information of the vehicle according to the target point tracked at the next moment.

[0020] S208. Construct an input of a vehicle trajectory tracking control strategy model based on the vehicle's observation information and the target reward function.

[0021] S209: Setting action constraints of the vehicle trajectory tracking control strategy model.

[0022] S210 , based on the input of the vehicle trajectory tracking control strategy model and the action constraints, a reinforcement learning algorithm is used to calculate the target reward function, thereby obtaining PID control parameters.

[0023] Optionally, obtaining the tracking error at the next moment based on the current vehicle state and the reference information of the predicted reference point in S204 includes:

[0024] According to the current state of the vehicle, the current PID controller parameters are obtained through reinforcement learning.

[0025] According to the current PID controller parameters, the output control quantity of the current PID controller is obtained.

[0026] According to the current state of the vehicle and the output control quantity of the PID controller at the current moment, the state of the vehicle at the next moment is obtained.

[0027] According to the vehicle state at the next moment and the reference information of the predicted reference point, the tracking error at the next moment is obtained.

[0028] Optionally, the target reward function in S205 is as shown in the following formula (1):

[0029] r(X,X ref ,u mv , Δa)=e(X,X ref ) T Pe(X,X ref )+u mv T Qu mv +Δa T RΔa (1)

[0030] In the formula, r(X, X ref ,u mv , Δa) represents the target reward function, X represents the state of the vehicle, X ref Represents the reference information of the predicted reference point, u mv =[ωsteer , a x ] T Represents the output control quantity of the PID controller, ω steer is the steering wheel angle, a x is the output longitudinal acceleration, Δa represents the change in strategy output, e(X, X ref ) represents the tracking error, T represents the matrix transpose, P represents the diagonal positive definite matrix used to control the tracking error, Q represents the diagonal positive definite matrix used to control the output control quantity, and R represents the diagonal positive definite matrix used to control the output change of the control strategy.

[0031] Optionally, the tracking error satisfies the following formula (2):

[0032]

[0033] Where y t+1 represents the longitudinal position of the vehicle in the ego-vehicle coordinate system at time t+1, Represents the ordinate in the reference information of the predicted reference point at time t+1, Indicates the heading angle of the vehicle in the ego vehicle coordinate system at time t+1, Indicates the heading angle in the reference information of the predicted reference point at time t+1, u t+1 It represents the longitudinal forward speed of the vehicle in the vehicle coordinate system at time t+1, represents the longitudinal forward speed in the reference information of the predicted reference point at time t+1, Δy t+1 represents the lateral error at time t+1, Indicates the heading angle error at time t+1, Δu t+1 Represents the velocity error at time t+1.

[0034] The strategy output change satisfies the following formula (3):

[0035]

[0036] Where, represents the proportional gain of the steering wheel angle PID controller at time t, represents the integral gain of the steering wheel angle PID controller at time t, represents the differential gain of the steering wheel angle PID controller at time t, represents the proportional gain of the longitudinal acceleration PID controller at time t, represents the integral gain of the longitudinal acceleration PID controller at time t, represents the differential gain of the longitudinal acceleration PID controller at time t, Indicates the proportional gain change of the angular velocity PID controller at time t+1, Indicates the change in the integral gain of the angular velocity PID controller at time t+1, Indicates the change in differential gain of the angular velocity PID controller at time t+1, It represents the proportional gain change of the longitudinal acceleration PID controller at time t+1, Indicates the change in the integral gain of the longitudinal acceleration PID controller at time t+1, Indicates the change in the differential gain of the longitudinal acceleration PID controller at time t+1.

[0037] Optionally, in S210 , based on the input of the vehicle trajectory tracking control strategy model and the action constraints, a reinforcement learning algorithm is used to calculate the target reward function, thereby obtaining PID control parameters, including:

[0038] According to the input of the vehicle trajectory tracking control strategy model, action constraints and reinforcement learning algorithm, the Actor-Critic structure is adopted to learn the value distribution objective function and strategy function, calculate the target reward function, and then obtain the PID control parameters.

[0039] Optionally, the PID controller in S4 includes: a lateral error PID controller and a speed error PID controller.

[0040] Among them, the lateral error PID controller is shown in the following formula (4):

[0041]

[0042] Where Δω steer Indicates the change in steering wheel angle, Δy t represents the lateral error at time t, Indicates the ratio of the steering wheel angle PID controller at time t, represents the integral of the steering wheel angle PID controller at time t, Represents the differential gain of the steering wheel angle PID controller at time t.

[0043] The speed error PID controller is shown in equation (5):

[0044]

[0045] Where Δa x Indicates the change in longitudinal acceleration, Δu t represents the longitudinal velocity error at time t, Represents the ratio of the longitudinal acceleration PID controller at time t, represents the integral of the longitudinal acceleration PID controller at time t, Represents the differential gain of the longitudinal acceleration PID controller at time t.

[0046] On the other hand, a PID trajectory tracking control device based on reinforcement learning is provided. The device is applied to the PID trajectory tracking control method based on reinforcement learning, and the device includes:

[0047] The acquisition module is used to obtain the current state, reference trajectory and actual operation error of the autonomous driving vehicle to be controlled.

[0048] Building module for constructing the strategy model of vehicle trajectory tracking control.

[0049] The input module is used to input the current state and reference trajectory into the vehicle trajectory tracking control strategy model to obtain PID control parameters.

[0050] The output module is used to input the PID control parameters and actual operation errors into the PID controller to obtain the control quantity of the autonomous driving vehicle, and complete the trajectory tracking control of the autonomous driving vehicle based on the control quantity of the autonomous driving vehicle.

[0051] On the other hand, a PID trajectory tracking control device is provided, which includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned PID trajectory tracking control methods based on reinforcement learning is implemented.

[0052] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned PID trajectory tracking control methods based on reinforcement learning.

[0053] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0054] In the embodiment of the present invention, a reinforcement learning algorithm is used to give coupling to multiple PID controls, thereby expanding the already widely used PID application to MIMO-related control tasks, and having a wide range of applications; the present invention solves the error amount based on reference information at the prediction moment, eliminates the lag of error feedback, and ensures timely and reliable control; the present invention retains the nonlinear characteristics of the controlled object model when constructing the control problem, and has high control accuracy; the present invention converts the online optimization problem into the offline solution of the policy network parameters by designing a loss function for offline training of the policy network, and does not need to store a large amount of state-to-control quantity mapping relationships, thereby saving memory resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0056] Figure 1 This is a flow chart of a PID trajectory tracking control method based on reinforcement learning provided by an embodiment of the present invention;

[0057] Figure 2 This is a schematic diagram of the overall process flow of the MIMO variable parameter PID trajectory tracking control method based on reinforcement learning provided by an embodiment of the present invention;

[0058] Figure 3 Schematic diagram of a MIMO variable parameter PID trajectory tracking control method based on reinforcement learning provided by an embodiment of the present invention;

[0059] Figure 4 1 is a schematic diagram (1) comparing trajectory tracking effects of the MIMO variable parameter PID trajectory tracking control method based on reinforcement learning provided by an embodiment of the present invention;

[0060] Figure 5 2 is a schematic diagram comparing the trajectory tracking effects of the MIMO variable parameter PID trajectory tracking control method based on reinforcement learning provided by an embodiment of the present invention;

[0061] Figure 6 This is a block diagram of a PID trajectory tracking control device based on reinforcement learning provided by an embodiment of the present invention;

[0062] Figure 7 It is a structural diagram of a PID trajectory tracking control device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0064] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0065] In the embodiments of the present invention, the terms "image" and "picture" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same. The terms "of," "corresponding," and "corresponding" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same.

[0066] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0067] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0068] The embodiment of the present invention provides a PID trajectory tracking control method based on reinforcement learning, which can be implemented by a PID trajectory tracking control device, which can be a terminal or a server. Figure 1 The flowchart of the PID trajectory tracking control method based on reinforcement learning is shown. The processing flow of this method may include the following steps:

[0069] First, there are several key terms in reinforcement learning that need to be explained:

[0070] 1. Environment: This refers to the external environment with which an agent interacts. It can be a physical environment in the real world, a virtual environment, or a simulated environment. The environment influences the state and actions of the agent and provides feedback based on the agent's actions.

[0071] 2. Agent: This refers to the subject involved in the learning and decision-making process. It can be a robot, a software program, or any entity capable of perception and decision-making. The agent observes the state of the environment and chooses appropriate actions to maximize rewards.

[0072] 3. State: Represents the current state or characteristics of the environment, which can be fully or partially observable. The agent selects an action based on the current state, and the environment reaches the next state based on the results of the action.

[0073] 4. Observation: This refers to the information the agent obtains about its state from the environment. In some cases, the state may not be directly observable, and the agent needs to approximate or infer the state through observation.

[0074] 5. Action: This refers to the behavior or decision that an agent can perform in a specific state. The agent chooses actions to maximize its reward.

[0075] 6. Reward: This represents the immediate feedback an agent receives after performing a specific action in a specific state. Rewards can be positive, negative, or zero, and are used to evaluate the quality of an action. The agent's goal is to maximize the cumulative reward by choosing actions.

[0076] 7. Objective Function: A value function constructed based on rewards that guides the agent’s learning and decision-making process. The agent’s goal is to improve its strategy by optimizing the reward-based value function to maximize the cumulative rewards.

[0077] By understanding and applying these terms, reinforcement learning algorithms enable intelligent agents to gradually improve their strategies in their interactions with the environment, thereby maximizing the total reward.

[0078] In the reinforcement learning of the present invention, the environment can be a system that simulates real-world conditions by performing state deduction on objects within it based on laws of physics, kinematics, and geometric constraints. This system consists of agents, obstacles, map constraints, and other components. Agents are physical elements in the environment that can act autonomously, receiving observations and taking actions, such as autonomous vehicles. States are basic properties of elements in the environment, such as the position, orientation, and velocity of a vehicle. Observations are encoded features of the state of other elements in the environment from the perspective of a given state. Actions are the outputs of the agent, determined by its policy. Deduction involves applying dynamic equations at each time step to iteratively calculate the state at each step, given a sequence of actions and the initial state of an object, ultimately producing a sequence of states as output. A state transition model is defined on the object. Given the object's current state and action, it outputs the state the object will transition to at the next step. This model is typically subject to constraints such as kinematic constraints. Rewards represent the evaluation of the state and action of elements in the environment at the current time step and are scalar values. Cumulative rewards are the sum of the rewards accumulated at each time step, starting from the current state, until the end of the deduction.

[0079] The vehicle trajectory tracking optimal control model is constructed within a reinforcement learning framework. The goal is to learn an optimal control strategy through interaction between the vehicle and the environment, maximizing the objective function obtained by executing this control strategy from the initial moment. The vehicle trajectory tracking control model consists of a policy network, a PID controller, and a state transition model. The policy network outputs the optimal PID control parameters for the current state based on the input vehicle state and reference trajectory. The PID controller outputs the vehicle control variable based on the current error information and the optimal parameters of the policy network. The state transition model outputs the vehicle state at the next moment, given the current vehicle state and the PID controller's control variable. This is determined by formulas such as the state-space equation.

[0080] S1. Obtain the current state, reference trajectory, and actual operating error of the autonomous driving vehicle to be controlled.

[0081] S2. Constructing a strategy model for vehicle trajectory tracking control. Optionally, the process of constructing the strategy model for vehicle trajectory tracking control in S2 may include S201-S210:

[0082] S201: Determine initial conditions and parameter initialization.

[0083] In a feasible implementation manner, the above step S201 is initialization of the environment, state and various parameters.

[0084] Specifically, at time t, the discrete interval and prediction time domain N of the vehicle trajectory tracking control system are determined, and the continuous reference trajectory, expected speed information and vehicle position state in the existing geodetic coordinate system are input into the vehicle trajectory tracking control system.

[0085] The reference trajectory information can be provided by the trajectory planning module; the expected speed information can be provided by the human-computer interaction module or the speed planning module; the vehicle position state can be provided by the state estimation and positioning module such as the odometer, for example, the vehicle state X0 = [0, 0, 0, 0, 0] at the initial moment. T .

[0086] Furthermore, the internal PID controller and reinforcement learning strategy are initialized.

[0087] Specifically, the PID controller sets the initial six-dimensional parameters for angular velocity and acceleration control At the same time, the reinforcement learning observations are initialized, creating a 3N+12-dimensional vector for subsequent data storage. Similarly, a 6-dimensional vector is created for the policy output to update the PID controller parameters in real time.

[0088] S202. After determining the initial conditions and initializing the parameters, obtain a continuous curve of the reference trajectory in the vehicle coordinate system described by a mathematical equation, determine the current tracking target point based on the continuous curve of the reference trajectory, and obtain the coordinates of the current tracking target point in the geodetic coordinate system.

[0089] In a feasible implementation manner, the above step S202 is to obtain the tracking target point at the current moment.

[0090] Specifically, at time t, after initialization, the continuous curve described by the mathematical equation of the reference trajectory point in the vehicle coordinate system is obtained, and the closest intersection point of the continuous curve with the reference trajectory point in the y-axis direction of the vehicle coordinate system is used as the current tracking target point, and the horizontal coordinate of the current tracking target point is converted into the horizontal coordinate of the current tracking target point in the geodetic coordinate by the following formula (1) (2) through the current vehicle state:

[0091]

[0092] Where x0 and y0 represent the horizontal and vertical coordinates of the target point being tracked at the current moment in geodetic coordinates. Represents the horizontal and vertical coordinates of the target point in the vehicle coordinate system at time t (according to the above tracking target point selection principle, ), Indicates the yaw angle measured at the current moment.

[0093] S203: Determine a prediction reference point based on the coordinates of the current tracking target point in geodetic coordinates, the discrete interval, the prediction time domain, and the preset prediction horizontal coordinate difference, and obtain reference information of the prediction reference point.

[0094] In a feasible implementation manner, the above step S203 is to calculate a reference quantity in the prediction time domain.

[0095] Specifically, in the geodetic coordinate system, the manually set discrete intervals and the preview horizontal coordinate differences manually set according to the desired preview effect are obtained. Taking x0 as the starting point, the horizontal coordinates of each preview reference point are obtained through the discrete intervals and the preview horizontal coordinate differences, and the various reference quantities such as the preview reference point vertical coordinate and yaw angle are calculated and stored in a queue.

[0096] For the reference trajectory of step S202, the horizontal coordinate of the preview reference point is obtained by the following formula:

[0097]

[0098] Where, It represents the horizontal coordinate of the kth preview reference point (prediction reference point), N is the prediction time domain, dx is the discrete interval (prediction horizontal coordinate interval), and x0 represents the horizontal coordinate of the current vehicle in the geodetic coordinate system.

[0099] Furthermore, according to the prediction time domain, we can obtain Compare x0 with Substitution Other reference quantities on the reference trajectory, such as the ordinate and heading angle, can be obtained.

[0100] For example, suppose that at t = 0, the current vehicle horizontal coordinate x0 = 0, the predicted horizontal coordinate interval is 0.5m, and the reference trajectory is y ref =sin(πx ref / 25), the prediction time domain is N=10; the horizontal coordinate in the reference information of the first prediction reference point is: The vertical axis is: The heading angle is:

[0101] S204: Obtain the tracking error at the next moment based on the current vehicle state and the reference information of the predicted reference point.

[0102] Optionally, the above step S204 may include the following steps S2041-S2044:

[0103] S2041. According to the current state of the vehicle, the current PID controller parameters are obtained through reinforcement learning.

[0104] S2042. Obtain the output control quantity of the PID controller at the current moment according to the PID controller parameters at the current moment.

[0105] S2043. Obtain the vehicle state at the next moment based on the vehicle state at the current moment and the output control variable of the PID controller at the current moment.

[0106] S2044: Obtain the tracking error at the next moment based on the vehicle state at the next moment and the reference information of the predicted reference point.

[0107] In a feasible implementation manner, the above step S204 is to obtain the vehicle state in discrete time.

[0108] Specifically, a vehicle coordinate system is established with the vehicle position at time t as the origin, and the vehicle state at time t+1 is obtained based on the vehicle dynamics equation and the output action of the trajectory tracking controller.

[0109] Among them, the vehicle state X at time t is t Contains the horizontal and vertical position x of the vehicle in the vehicle coordinate system at time t t ,y t , heading angle Longitudinal forward speed u t , lateral sideslip speed v t and angular velocity ωt ,Right now

[0110] Furthermore, the strategy output at time t includes the following: t Feedback and speed error Δu t Feedback PID controller parameters Make the PID controller output the relevant control quantity u mv =[ω steer , a x ] T To achieve coupling control; where ω steer is the steering wheel angle, and ax is the output longitudinal acceleration.

[0111] S205. Construct a target reward function based on the tracking error at the next moment.

[0112] In a feasible implementation, the above step S205 is to construct a target reward function.

[0113] Specifically, the target reward function is constructed using the tracking error at time t+1 obtained in step S204.

[0114] Among them, the reward function r(X, X ref ,u mv , Δa) is designed as the tracking error e(X, X ref ), control amount u mv , the quadratic weighted sum of the strategy output change Δa, that is:

[0115] r(X,X ref ,u mv , Δa)=e(X,X ref ) T Pe(X,X ref )+u mv T Qu mv +Δa T RΔa (4)

[0116] In the formula, r(X, X ref ,u mv , Δa) represents the target reward function, X represents the state of the vehicle, X ref Represents the reference information of the predicted reference point, u mv =[ω steer , a x ] T Represents the output control quantity of the PID controller, ω steer is the steering wheel angle, a x is the output longitudinal acceleration, Δa represents the change in strategy output, e(X, X ref) represents the tracking error, T represents the matrix transpose, P, Q, and R are all diagonal positive definite matrices, which are used to control the weight of the tracking error and the influence of the control quantity on the control result.

[0117] Among them, the state tracking error satisfies the following formula (5):

[0118]

[0119] Where y t+1 represents the longitudinal position of the vehicle in the ego-vehicle coordinate system at time t+1, Represents the ordinate in the reference information of the predicted reference point at time t+1, Indicates the heading angle of the vehicle in the ego vehicle coordinate system at time t+1, Indicates the heading angle in the reference information of the predicted reference point at time t+1, u t+1 It represents the longitudinal forward speed of the vehicle in the vehicle coordinate system at time t+1, represents the longitudinal forward speed in the reference information of the predicted reference point at time t+1, Δy t+1 represents the lateral error at time t+1, Indicates the heading angle error at time t+1, Δu t+1 Represents the velocity error at time t+1.

[0120] For the action fluctuation, the following formula (6) is satisfied:

[0121]

[0122] Where, represents the proportional gain of the steering wheel angle PID controller at time t, represents the integral gain of the steering wheel angle PID controller at time t, represents the differential gain of the steering wheel angle PID controller at time t, represents the proportional gain of the longitudinal acceleration PID controller at time t, represents the integral gain of the longitudinal acceleration PID controller at time t, represents the differential gain of the longitudinal acceleration PID controller at time t, Indicates the proportional gain change of the angular velocity PID controller at time t+1, Indicates the change in the integral gain of the angular velocity PID controller at time t+1, Indicates the change in differential gain of the angular velocity PID controller at time t+1, It represents the proportional gain change of the longitudinal acceleration PID controller at time t+1, Indicates the change in the integral gain of the longitudinal acceleration PID controller at time t+1, Indicates the change in the differential gain of the longitudinal acceleration PID controller at time t+1.

[0123] S206: Obtain the tracking target point at the next moment according to the output control quantity of the PID controller at the current moment.

[0124] In a feasible implementation manner, the above step S206 is updating the reference trajectory points.

[0125] Specifically, based on the vehicle dynamics equation, the PID controller output u at the previous moment is mv The calculated horizontal coordinate x of the vehicle at time t+1 0,t+1 , obtain a new reference point information sequence.

[0126] S207: Acquire the observation information of the vehicle according to the target point tracked at the next moment.

[0127] In a feasible implementation manner, the above step S207 is to obtain control model observations.

[0128] Specifically, the vehicle's observations consist of the following four parts: the vehicle's own observations X t , predicted trajectory reference information X ref , error tracking observation E t and the strategy output a at the previous moment t-1 .

[0129] Among them, the vehicle observation quantity X t The information can be obtained in step S203, and the predicted trajectory reference information X ref Can be obtained in step S205, including reference information of N moments k∈1,2...N; error tracking observation

[0130] E t =[Δy t -Δy t-1 , Δy t , Δy t -2Δy t-1 +Δy t-2 , Δu t -Δu t-1 , Δu t , Δu t -2Δu t-1 +Δu t-2 ] T And the strategy output at the previous moment It can be obtained in step S205.

[0131] S208. Construct an input of a vehicle trajectory tracking control strategy model based on the vehicle's observation information and the target reward function.

[0132] In a feasible implementation, the observation information obtained in step S207 and the reward function in step S205 are used as inputs of the vehicle trajectory tracking control model, and the observation information is input into the control model.

[0133] S209: Setting action constraints of the vehicle trajectory tracking control strategy model.

[0134] In a feasible implementation, in order to ensure the stability of vehicle control, the action is set to a bounded output, and finally the vehicle state space equation with control quantity constraints can be obtained:

[0135]

[0136] Among them, A is the state matrix, B is the input matrix, X k represents the state of the vehicle at time k, and a and b are the lower and upper bounds of the control quantity.

[0137] S210. Based on the input of the vehicle trajectory tracking control strategy model, action constraints, and reinforcement learning algorithm, an Actor-Critic structure is used to learn the value distribution objective function and the strategy function, calculate the target reward function, and then obtain the PID control parameters.

[0138] The value distribution reinforcement learning algorithm uses an actor-critic structure to learn independent value distribution functions and policy functions. The mathematical expression of the value distribution objective function is as follows:

[0139]

[0140] u=π θ (X, X ref ;ω) (9)

[0141] Z(X,X ref ,u)=r+γG t+1 (10)

[0142]

[0143] In formula (8), Qz is the target value function of vehicle trajectory tracking control, X, X ref and u are the state, reference trajectory, and action of the ego vehicle at any time in the infinite time domain, θ is the parameter to be optimized in the parameterized target value network, and B is the experience pool used to store the training samples (X, X) generated by the interaction between the strategy and the environment. ref ,u,r,X′,X′ ref ), r is the reward function, is the Bellman self-consistent operator, and θ′ represents the parameters of the old value network before updating.

[0144] In formula (9), π θ (X, X ref ω) is a parameterized policy network, which inputs the vehicle state X and the environment state X at any time ref The action u of the vehicle can be obtained, ω is the parameter to be optimized in the parameterized strategy, and u′ represents the action of the vehicle by another strategy network π ω′ Output action.

[0145] In formula (10), formula (11), and formula (12), Z is the cumulative reward generated by the flexible strategy π, which is the state-action reward, denoted as Zσ(X, X ref ,u),γ is the discount factor; is the probability density function of the flexible Z value, G t represents the random flexible return of strategy π, α represents the strategy entropy temperature coefficient, σ Z represents the variance of the cumulative return Z.

[0146] Strategy function, where the mathematical expression of the strategy objective function is as follows:

[0147]

[0148] Among them, Q is given by the value distribution function as the policy entropy temperature coefficient, and its update rule is as follows:

[0149]

[0150] Among them, β c is the learning rate, is the given policy entropy target value.

[0151] like Figure 2 、 Figure 3 As shown in Figure 2, in the specific implementation process, the reinforcement learning part of the vehicle trajectory tracking control needs to be solved iteratively, which specifically includes the following steps:

[0152] Step 10-1: Give relevant initial conditions and parameter initialization.

[0153] The initial state of the vehicle can be given directly, such as X0 = [0, 0, 0, 0] T , can also be randomly generated within a reasonable range; given the reference trajectory y ref =sin(πx ref \25), with reference to the expected speed u ref =5m / s, prediction interval dx=0.5m, prediction time domain N=10; initialization time t, given experience pool B size is 10 6 And randomly initialized to store the training samples (X, X) generated by the interaction between the strategy and the environment ref,u,r,X′,X′ ref ); initialize the value distribution network parameters θ, policy network parameters ω and policy entropy coefficient α; initialize the target network parameters θ′←θ, ω′←ω; give a suitable learning rate β a =10 -5 , β b =10 -5 ,τ=10 -5 and error∈=10 -5 Initialize the iteration number k = 0. The value distribution network and the policy network both use fully connected multi-layer perceptrons with a hidden layer size of 256×256.

[0154] Step 10-2: Obtain reference quantities within the prediction time domain.

[0155] The N reference trajectory points in the geodetic coordinate system in the prediction time domain are obtained by using the horizontal coordinate interval length dx and the prediction step length N in discrete time, which are

[0156] Step 10-3: Sampling.

[0157] Loop index variable k = k + 1, in the current state X t Using strategy π ω Interact with the environment to sample and obtain a reward function and observe the next state X t+1 , and obtain a new reference point X t ,u t , R t+1 , X t+1 , Form an experience sample and store the samples into experience pool B one by one.

[0158] Step 10-4: Calculate the value of the value distribution objective function.

[0159] The multiple experience samples sampled from the experience replay pool B are used as the initial values ​​of the forward solution process. The value distribution network shown in formula (8) is used to calculate the objective function values ​​respectively, providing initial values ​​for the backward solution value distribution network gradient and parameter update.

[0160] Step 10-5: Update the value distribution network parameters.

[0161] The gradient of the objective function value obtained is calculated, and after averaging the gradient of the batch-calculated network, the network parameters are subjected to gradient descent. The update rule of the median network in the kth iteration is:

[0162]

[0163] Where, represents the θ gradient operator, θk Represents the objective function value in the kth iteration.

[0164] Step 10-6: Update policy network parameters.

[0165] If k is a positive integer multiple of m, then use Equation (15) to batch calculate the objective function and solve the policy gradient, and use the averaged policy gradient to perform gradient descent on the policy parameters. The update rule of the policy network in the Kth iteration is:

[0166]

[0167] Where, β b Indicates the learning rate, represents the ω gradient operator.

[0168] Step 10-8: Update the policy entropy coefficient.

[0169] If k is a positive integer multiple of m, update is performed using formula (14).

[0170] Step 10-9: Update the target value network and strategy network parameters.

[0171] If k is a positive integer multiple of m, then update according to the following rules:

[0172] θ′ k+1 =-(1-τ)θ k ′+τθ k (18)

[0173] ω′ k+1 =-(1-τ)ω k ′+τω k (19)

[0174] Step 10-10: Determine whether the result has converged.

[0175] If |J π (ω k+1 )-J π (ω k )|>∈, it indicates that the strategy has not converged, and return to step 10-3 to continue solving; otherwise, it indicates that the strategy has converged within the error range, and the output is the optimal parameterized strategy.

[0176] S211. Output network results.

[0177] The parameterized network is used as the output of the policy network. The policy network can output the tuning parameters of the vehicle trajectory tracking coupled PID control model based on the vehicle state and prediction information.

[0178] S3. Input the current state and reference trajectory into the vehicle trajectory tracking control strategy model to obtain PID control parameters.

[0179] S4. Input the PID control parameters and the actual operation error into the PID controller to obtain the control quantity of the autonomous driving vehicle. According to the control quantity of the autonomous driving vehicle, the trajectory tracking control of the autonomous driving vehicle is completed.

[0180] In a feasible implementation, the output of step S210 is The input is fed into two coupled PID controllers for lateral error and velocity error feedback to achieve MIMO coupled control. The mathematical expressions of the lateral error and velocity error PID controllers are as follows:

[0181]

[0182] It is worth noting that equations (20) and (21) provide feedback based on the prediction error at time t+1 obtained in step S203, eliminating the hysteresis of error feedback and improving control accuracy and effect.

[0183] The present invention adopts a reinforcement learning algorithm and utilizes a parameterized strategy network obtained through offline training to directly output the optimal PID parameters of the vehicle according to the current state of the vehicle and predicted reference information. This enables the PID control output under the coupling parameters to realize the change of the vehicle state space, and continuously outputs the control quantity according to the vehicle state and environmental information, thereby realizing high real-time and high-precision trajectory tracking control of the vehicle.

[0184] The present invention uses the optimal parameterized strategy network learned after 400,000 iterations of the vehicle trajectory tracking control model to simulate and control the vehicle to u ref =5m / s expected longitudinal speed tracking the given reference curve, the results obtained under the same reference trajectory and reference speed are compared with the results of uncoupled PID control, such as Figure 4 、 Figure 5 As shown in the figure, the results show that the method proposed in the present invention achieves good tracking effect for both position and velocity in the entire trajectory tracking process. Compared with the PID control under non-coupling, the ordinary PID exposes the problem of control lag in the control, resulting in poor tracking effect at the sudden change of trajectory corners, and in speed control, weak oscillation is re-generated after convergence due to the coupling problem. However, the method of the present invention solves the above problems better due to the introduction of preview error and reinforcement learning.

[0185] This paper proposes a method for constructing and solving variable-parameter PID trajectory tracking control based on reinforcement learning. First, the entire control system identifies the closest point between the vehicle's lateral position and the reference trajectory, assuming a given desired speed and reference trajectory, and thus determines the vehicle's current and expected optimal positions. Next, a reinforcement learning algorithm outputs PID coupling parameters. These parameters are input into a PID controller, which predicts the vehicle's subsequent state based on its discrete dynamic characteristics, achieving trajectory tracking control.

[0186] The core of the present invention is to use the vehicle's current state, expected trajectory information, and lateral deviation sets at different time points as input data, and through the network strategy learned offline through reinforcement learning, dynamically adjust the various PID parameters, and finally output the control quantity by each PID controller. This step transforms the complex online optimization problem of the common trajectory tracking algorithm into a policy network parameter problem that can be solved offline through reinforcement learning. When applied online, the on-board control system can directly output and adjust the optimal PID control parameters through the policy network in real time according to the current state of the vehicle and the actual operating error, and then achieve high robustness control of the vehicle behavior through PID control. This process ensures that the PID controller can achieve high-precision control effects of multi-variable coupling while ensuring stability and reliability, and realize efficient online real-time trajectory tracking optimization.

[0187] The present invention has the advantages of high computational efficiency (the strategy function only needs to approximately fit the optimal parameters), strong reliability (the PID controller itself has good anti-interference ability), high precision (the strategy learning process retains the nonlinear characteristics of the controlled object model), and a wide range of applications (the method takes the simultaneous elimination of multiple coupling errors as the control goal and is not limited to trajectory tracking tasks). It breaks through the complexity of implementing MIMO coupling control with PID and the problems of poor online solution efficiency of existing trajectory tracking control problems, complex design and implementation rules, and reliability that needs to be strictly proven.

[0188] In the embodiment of the present invention, a reinforcement learning algorithm is used to give coupling to multiple PID controls, thereby expanding the already widely used PID application to MIMO-related control tasks, and having a wide range of applications; the present invention solves the error amount based on reference information at the prediction moment, eliminates the lag of error feedback, and ensures timely and reliable control; the present invention retains the nonlinear characteristics of the controlled object model when constructing the control problem, and has high control accuracy; the present invention converts the online optimization problem into the offline solution of the policy network parameters by designing a loss function for offline training of the policy network, and does not need to store a large amount of state-to-control quantity mapping relationships, thereby saving memory resources.

[0189] Figure 6This is a block diagram of a PID trajectory tracking control device based on reinforcement learning according to an exemplary embodiment. The device is used for a PID trajectory tracking control method based on reinforcement learning. Figure 6 The device includes an acquisition module 310, a construction module 320, an input module 330 and an output module 340.

[0190] in:

[0191] The acquisition module 310 is used to obtain the current state, reference trajectory and actual operation error of the autonomous driving vehicle to be controlled.

[0192] The construction module 320 is used to construct a strategy model for vehicle trajectory tracking control.

[0193] The input module 330 is used to input the current state and the reference trajectory into the vehicle trajectory tracking control strategy model to obtain PID control parameters.

[0194] The output module 340 is used to input the PID control parameters and the actual operation error into the PID controller to obtain the control quantity of the autonomous driving vehicle, and complete the trajectory tracking control of the autonomous driving vehicle based on the control quantity of the autonomous driving vehicle.

[0195] In the embodiment of the present invention, a reinforcement learning algorithm is used to give coupling to multiple PID controls, thereby expanding the already widely used PID application to MIMO-related control tasks, and having a wide range of applications; the present invention solves the error amount based on reference information at the prediction moment, eliminates the lag of error feedback, and ensures timely and reliable control; the present invention retains the nonlinear characteristics of the controlled object model when constructing the control problem, and has high control accuracy; the present invention converts the online optimization problem into the offline solution of the policy network parameters by designing a loss function for offline training of the policy network, and does not need to store a large amount of state-to-control quantity mapping relationships, thereby saving memory resources.

[0196] Figure 7 : is a structural diagram of a PID trajectory tracking control device provided by an embodiment of the present invention, such as Figure 7 As shown, the PID trajectory tracking control device may include the above Figure 6 The PID trajectory tracking control device based on reinforcement learning is shown. Optionally, the PID trajectory tracking control device 410 may include a first processor 2001.

[0197] Optionally, the PID trajectory tracking control device 410 may further include a memory 2002 and a transceiver 2003 .

[0198] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0199] The following combination Figure 7 The components of the PID trajectory tracking control device 410 are described in detail:

[0200] The first processor 2001 is the control center of the PID trajectory tracking control device 410 and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs) or one or more field programmable gate arrays (FPGAs).

[0201] Optionally, the first processor 2001 may execute various functions of the PID trajectory tracking control device 410 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0202] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 7 CPU0 and CPU1 are shown in FIG.

[0203] In a specific implementation, as an embodiment, the PID trajectory tracking control device 410 may also include multiple processors, such as Figure 7 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0204] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0205] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and be controlled by the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0206] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0207] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 7 (not shown separately in the figure). The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0208] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and track the interface circuit of the control device 410 through the PID trajectory ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0209] It should be noted that Figure 7 The structure of the PID trajectory tracking control device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0210] In addition, the technical effects of the PID trajectory tracking control device 410 can refer to the technical effects of the PID trajectory tracking control method based on reinforcement learning described in the above method embodiment, and will not be repeated here.

[0211] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0212] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0213] The above embodiments can be implemented in whole or in part through software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0214] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0215] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0216] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0217] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0218] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0219] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0220] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0221] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0222] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0223] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A PID trajectory tracking control method based on reinforcement learning, characterized in that: The method comprises: S1. Obtain the current state, reference trajectory, and actual operating error of the autonomous driving vehicle to be controlled; S2. Construct a strategy model for vehicle trajectory tracking control; S3, inputting the current state and reference trajectory into a vehicle trajectory tracking control strategy model to obtain PID control parameters; S4. Inputting the PID control parameters and the actual operation error into a PID controller to obtain a control variable of the autonomous driving vehicle, and completing trajectory tracking control of the autonomous driving vehicle based on the control variable of the autonomous driving vehicle; The process of constructing the vehicle trajectory tracking control strategy model in S2 includes: S201, determining initial conditions and parameter initialization; The determining of the initial conditions includes: determining the discrete interval and the prediction time domain, obtaining the reference trajectory, the expected speed information and the vehicle state; The parameter initialization includes: initializing the PID controller and the reinforcement learning strategy; S202: After determining initial conditions and initializing parameters, obtaining a continuous curve of a reference trajectory in a vehicle coordinate system described by a mathematical equation, determining a current tracking target point based on the continuous curve of the reference trajectory, and obtaining the coordinates of the current tracking target point in geodetic coordinates; S203: Determine a prediction reference point based on the coordinates of the current tracking target point in geodetic coordinates, the discrete interval, the prediction time domain, and the preset prediction horizontal coordinate difference, and obtain reference information of the prediction reference point; S204: Obtain the tracking error at the next moment based on the current vehicle state and the reference information of the predicted reference point; S205: Constructing a target reward function based on the tracking error at the next moment; S206, obtaining the tracking target point at the next moment based on the output control quantity of the PID controller at the current moment; S207, obtaining observation information of the vehicle according to the next moment tracking target point; S208: constructing an input of a vehicle trajectory tracking control strategy model based on the vehicle's observation information and the target reward function; S209, setting the action constraint conditions of the vehicle trajectory tracking control strategy model; S210 : Based on the input of the vehicle trajectory tracking control strategy model and the action constraints, a reinforcement learning algorithm is used to calculate the target reward function, thereby obtaining PID control parameters.

2. The PID trajectory tracking control method based on reinforcement learning according to claim 1 is characterized in that: The step S204 of obtaining the tracking error at the next moment based on the current vehicle state and the reference information of the predicted reference point includes: According to the current state of the vehicle, the current PID controller parameters are obtained through reinforcement learning; According to the current PID controller parameters, the output control amount of the current PID controller is obtained; Obtaining the vehicle state at the next moment according to the vehicle state at the current moment and the output control variable of the PID controller at the current moment; The tracking error at the next moment is obtained according to the vehicle state at the next moment and the reference information of the predicted reference point.

3. The PID trajectory tracking control method based on reinforcement learning according to claim 1 is characterized in that: The target reward function in S205 is shown in the following formula (1): r(X,X ref ,u mv ,Δa)=e(X,X ref ) T For(X,X ref )+u mv T Qu mv +Δa T RΔa (1) In the formula, r(X, X ref ,u mv , Δa) represents the target reward function, X represents the state of the vehicle, X ref Represents the reference information of the predicted reference point, u mv =[ω steer , a x ] T Represents the output control quantity of the PID controller, ω steer is the steering wheel angle, a x is the output longitudinal acceleration, Δa represents the change in strategy output, e(X,X ref ) represents the tracking error, T represents the matrix transpose, P represents the diagonal positive definite matrix used to control the tracking error, Q represents the diagonal positive definite matrix used to control the output control quantity, and R represents the diagonal positive definite matrix used to control the output change of the control strategy.

4. The PID trajectory tracking control method based on reinforcement learning according to claim 3 is characterized in that: The tracking error satisfies the following formula (2): Where y t+1 represents the longitudinal position of the vehicle in the ego-vehicle coordinate system at time t+1, Represents the ordinate in the reference information of the predicted reference point at time t+1, Indicates the heading angle of the vehicle in the ego-vehicle coordinate system at time t+1, Indicates the heading angle in the reference information of the predicted reference point at time t+1, u t+1 It represents the longitudinal forward speed of the vehicle in the vehicle coordinate system at time t+1, represents the longitudinal forward speed in the reference information of the predicted reference point at time t+1, Δy t+1 represents the lateral error at time t+1, Indicates the heading angle error at time t+1, Δu t+1 represents the velocity error at time t+1; The strategy output variation satisfies the following formula (3): Where, represents the proportional gain of the steering wheel angle PID controller at time t, represents the integral gain of the steering wheel angle PID controller at time t, represents the differential gain of the steering wheel angle PID controller at time t, represents the proportional gain of the longitudinal acceleration PID controller at time t, represents the integral gain of the longitudinal acceleration PID controller at time t, represents the differential gain of the longitudinal acceleration PID controller at time t, Indicates the proportional gain change of the angular velocity PID controller at time t+1, Indicates the change in the integral gain of the angular velocity PID controller at time t+1, Indicates the change in differential gain of the angular velocity PID controller at time t+1, It represents the proportional gain change of the longitudinal acceleration PID controller at time t+1, Indicates the change in the integral gain of the longitudinal acceleration PID controller at time t+1, Indicates the change in the differential gain of the longitudinal acceleration PID controller at time t+1.

5. The PID trajectory tracking control method based on reinforcement learning according to claim 1, characterized in that: The step S210 uses a reinforcement learning algorithm to calculate the target reward function based on the input of the vehicle trajectory tracking control strategy model and the action constraints, thereby obtaining PID control parameters, including: According to the input of the vehicle trajectory tracking control strategy model, action constraints and reinforcement learning algorithm, an Actor-Critic structure is adopted to learn the value distribution objective function and strategy function, calculate the target reward function, and then obtain the PID control parameters.

6. The PID trajectory tracking control method based on reinforcement learning according to claim 1, characterized in that: The PID controller in S4 includes: a lateral error PID controller and a speed error PID controller; The lateral error PID controller is as shown in the following formula (4): Where Δω steer Indicates the change in steering wheel angle, Δy t represents the lateral error at time t, represents the proportional gain of the steering wheel angle PID controller at time t, represents the integral gain of the steering wheel angle PID controller at time t, represents the differential gain of the steering wheel angle PID controller at time t; The speed error PID controller is shown in the following formula (5): Where Δa x Indicates the change in longitudinal acceleration, Δu t represents the longitudinal velocity error at time t, represents the proportional gain of the longitudinal acceleration PID controller at time t, represents the integral gain of the longitudinal acceleration PID controller at time t, Represents the differential gain of the longitudinal acceleration PID controller at time t.

7. A PID trajectory tracking control device based on reinforcement learning, wherein the PID trajectory tracking control device based on reinforcement learning is used to implement the PID trajectory tracking control method based on reinforcement learning according to any one of claims 1 to 6, characterized in that: The device comprises: An acquisition module is used to obtain the current state, reference trajectory and actual operation error of the autonomous driving vehicle to be controlled; Building module, used to build a strategy model for vehicle trajectory tracking control; An input module, configured to input the current state and reference trajectory into a vehicle trajectory tracking control strategy model to obtain PID control parameters; The output module is used to input the PID control parameters and the actual operation error into the PID controller to obtain the control quantity of the autonomous driving vehicle, and complete the trajectory tracking control of the autonomous driving vehicle according to the control quantity of the autonomous driving vehicle.

8. A PID trajectory tracking control device, characterized in that: The PID trajectory tracking control device includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images