Vehicle control method, system and equipment based on reinforcement learning and medium

Abstract

The invention provides a vehicle control method, system and device based on reinforcement learning and a medium. The method comprises the steps that vehicle state information and reference track information of a target vehicle at the current moment are acquired; performing rolling optimization solution through a model prediction control algorithm to obtain a control sequence of a coverage control time domain corresponding to the current moment; constructing a strategy network for inputting the reinforcement learning state vector into a pre-trained flexible action-evaluation network to obtain a front wheel steering angle correction amount at the current moment; and carrying out superposition processing on the first-item front-wheel rotation angle increment, the front-wheel rotation angle correction and the front-wheel rotation angle control quantity at the previous moment in the control sequence to generate the front-wheel rotation angle control quantity at the current moment. According to the method, the controller does not completely depend on a fixed vehicle dynamics model any more, so that the technical problem that the performance is reduced or even unstable due to the fact that the model is inaccurate in an existing algorithm is effectively solved, and high-precision trajectory tracking can still be kept under the complex and changeable real driving working condition.

Description

Technical Field

[0001] This invention application relates to the field of vehicle assisted driving, and more particularly to a vehicle control method, system, device and medium based on reinforcement learning. Background Technology

[0002] Currently, in vehicle control for scenarios such as assisted driving and autonomous driving, Model Predictive Control (MPC) and Linear Quadratic Regulator (LQR) algorithms are commonly used to achieve path tracking and vehicle control. MPC uses a vehicle dynamics model to predict the system's behavior over a future period (i.e., the prediction time domain). It finds a control sequence that optimizes future performance metrics (cost functions) by solving a constrained optimization problem. However, its control performance heavily relies on the accuracy of its internal prediction model, making it difficult to handle the nonlinearity and uncertainty of real-world operating conditions. When the actual dynamic characteristics of the vehicle deviate from the fixed, simplified prediction model used by the controller, the performance of the MPC algorithm drops sharply, potentially leading to vehicle instability. Therefore, ensuring the accuracy of vehicle control has become a critical issue that needs to be addressed. Summary of the Invention

[0003] This invention application provides a vehicle control method, system, device, and medium based on reinforcement learning to solve the technical problem of how to improve the accuracy of vehicle control and reduce the risk of vehicle instability.

[0004] To address the aforementioned technical problems, this invention provides a vehicle control method based on reinforcement learning, comprising: Obtain the current vehicle status information and reference trajectory information of the target vehicle; wherein, the current vehicle status information includes the current vehicle position and the current heading angle; Based on the vehicle state information and reference trajectory information, a rolling optimization solution is performed using a model predictive control algorithm to obtain the control sequence covering the control time domain at the current moment; wherein, the control sequence is the front wheel steering angle increment sequence; A reinforcement learning state vector is constructed based on the vehicle state information, reference trajectory information, and control sequence; and the reinforcement learning state vector is input into the policy network of a pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction at the current moment. The front wheel angle increment, front wheel angle correction, and front wheel angle control quantity from the previous moment are superimposed in the control sequence to generate the front wheel angle control quantity at the current moment.

[0005] As a preferred embodiment, the step of inputting the reinforcement learning state vector into the policy network of a pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction at the current moment includes: The reinforcement learning state vector is input into the policy network of a pre-trained flexible action-evaluation network; Based on the output of the policy network, the corresponding Gaussian distribution is obtained; The Gaussian distribution is compressed using the hyperbolic tangent function to obtain the action correction base value; The action correction base value is corrected by using the preset maximum correction range of the front wheel angle to obtain the front wheel angle correction amount at the current moment.

[0006] As a preferred embodiment, before inputting the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network, the following steps are included: A preset experience replay pool is sampled to obtain a certain number of training data; wherein each piece of training data includes a state vector at a historical moment; Using the training data, a basic network model to be trained is trained; wherein, the basic network model includes a target Q-network and a main Q-network; During each training session, the soft Bellman target value for the next time step of the historical moment is calculated using the target Q-network. The soft Bellman objective value is used as a supervision signal. The parameters of the main Q network are updated by minimizing the mean square error loss. The parameters of the policy network are then updated using the output of the updated main Q network. When the preset convergence condition is met, the trained flexible action-evaluation network is obtained.

[0007] As a preferred embodiment, the step of compressing the Gaussian distribution using the hyperbolic tangent function to obtain the action correction base value specifically involves: The Gaussian distribution is compressed according to the following formula to obtain the action correction base value: ; in, The action correction value at time t. Let s be a random noise vector sampled from a standard normal distribution. t Let be the reinforcement learning state vector. The mean of the Gaussian distribution is given. Let be the standard deviation of the Gaussian distribution.

[0008] As a preferred embodiment, the reference trajectory information includes the vehicle reference position and reference heading angle of the predicted data point closest to the target vehicle in the reference trajectory; Based on the vehicle state information and reference trajectory information, a rolling optimization solution is performed using a model predictive control algorithm to obtain the control sequence corresponding to the current time-domain coverage control, including: Based on the vehicle reference position and the current vehicle position, calculate the lateral deviation and lateral deviation rate of change of the target vehicle; based on the reference heading angle and the current heading angle, calculate the heading deviation and heading deviation rate of change of the target vehicle. Substitute the lateral deviation, lateral deviation rate of change, heading deviation, and heading deviation rate of change into the preset first state space equation corresponding to the target vehicle and perform rolling solution to obtain the sequence to be optimized. The control sequence is obtained by using a model predictive control algorithm to optimize the sequence to be optimized.

[0009] As a preferred embodiment, before substituting the lateral deviation, lateral deviation rate of change, heading deviation, and heading deviation rate of change into the preset first state space equation corresponding to the target vehicle for rolling solution, the method further includes: Obtain the vehicle parameters of the target vehicle; wherein the vehicle parameters include the vehicle's curb weight, front axle side stiffness, rear axle side stiffness, moment of inertia about the vertical axis, distance from the center of mass to the front axle, and distance from the center of mass to the rear axle. Based on the vehicle parameters, construct a state-space expression; The state-space expression is discretized to obtain the first state-space equation.

[0010] As a preferred embodiment, the step of employing a model predictive control algorithm to optimize the sequence to obtain the control sequence includes: Under preset constraints, the sequence to be optimized is obtained by optimizing the objective function of the model predictive control algorithm to obtain the control sequence; wherein... The objective function is to minimize the result calculated according to the following cost function: ; Where J is the cost function, Y is the state value sequence matrix in the prediction time domain, and Y ref Let U be the expected state value sequence matrix in the prediction time domain, and ΔU be the sequence to be optimized. k-1 H is the control sequence matrix from the previous time step, where H is U. k-1 The lower triangular matrix obtained by transforming and solving between △U, As slack variables, Here, represents the weighting coefficients of the slack variables, Q represents the weighting matrix of the output error, R represents the weighting matrix of the control increment, and S represents the weighting matrix of the control quantity.

[0011] Accordingly, this invention application also provides a vehicle control system based on reinforcement learning, including an information acquisition module, a solution module, a correction module, and a generation module; wherein, The information acquisition module is used to acquire the vehicle status information and reference trajectory information of the target vehicle at the current moment; wherein, the vehicle status information at the current moment includes the current vehicle position and the current heading angle; The solution module is used to perform rolling optimization based on the vehicle state information and reference trajectory information using a model predictive control algorithm to obtain the control sequence covering the control time domain at the current moment; wherein, the control sequence is the front wheel steering angle increment sequence; The correction module is used to construct a reinforcement learning state vector based on the vehicle state information, reference trajectory information and control sequence, and input the reinforcement learning state vector into the policy network of a pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction amount at the current moment. The generation module is used to superimpose the first front wheel angle increment, the front wheel angle correction, and the front wheel angle control quantity of the previous moment in the control sequence to generate the front wheel angle control quantity of the current moment.

[0012] As a preferred embodiment, the correction module inputs the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction amount at the current moment, including: The correction module inputs the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network; Based on the output of the policy network, the corresponding Gaussian distribution is obtained; The Gaussian distribution is compressed using the hyperbolic tangent function to obtain the action correction base value; The action correction base value is corrected by using the preset maximum correction range of the front wheel angle to obtain the front wheel angle correction amount at the current moment.

[0013] As a preferred embodiment, the vehicle control system further includes a training module, which is used before the correction module inputs the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network: A preset experience replay pool is sampled to obtain a certain number of training data; wherein each piece of training data includes a state vector at a historical moment; Using the training data, a basic network model to be trained is trained; wherein, the basic network model includes a target Q-network and a main Q-network; During each training session, the soft Bellman target value for the next time step of the historical moment is calculated using the target Q-network. The soft Bellman objective value is used as a supervision signal. The parameters of the main Q network are updated by minimizing the mean square error loss. The parameters of the policy network are then updated using the output of the updated main Q network. When the preset convergence condition is met, the trained flexible action-evaluation network is obtained.

[0014] As a preferred embodiment, the correction module compresses the Gaussian distribution using a hyperbolic tangent function to obtain the action correction base value, specifically: The correction module compresses the Gaussian distribution according to the following formula to obtain the action correction base value: ; in, The action correction value at time t. Let s be a random noise vector sampled from a standard normal distribution. t Let be the reinforcement learning state vector. The mean of the Gaussian distribution is given. Let be the standard deviation of the Gaussian distribution.

[0015] As a preferred embodiment, the reference trajectory information includes the vehicle reference position and reference heading angle of the predicted data point closest to the target vehicle in the reference trajectory; The information acquisition module, based on the vehicle state information and reference trajectory information, performs rolling optimization using a model predictive control algorithm to obtain the control sequence covering the control time domain corresponding to the current moment, including: The information acquisition module calculates the lateral deviation and lateral deviation rate of change of the target vehicle based on the vehicle reference position and the current vehicle position; and calculates the heading deviation and heading deviation rate of change of the target vehicle based on the reference heading angle and the current heading angle. Substitute the lateral deviation, lateral deviation rate of change, heading deviation, and heading deviation rate of change into the preset first state space equation corresponding to the target vehicle and perform rolling solution to obtain the sequence to be optimized. The control sequence is obtained by using a model predictive control algorithm to optimize the sequence to be optimized.

[0016] As a preferred embodiment, the vehicle control system further includes an equation construction module, which is used before the information acquisition module substitutes the lateral deviation, lateral deviation rate of change, heading deviation, and heading deviation rate of change into the preset first state space equation corresponding to the target vehicle for rolling solution: Obtain the vehicle parameters of the target vehicle; wherein the vehicle parameters include the vehicle's curb weight, front axle side stiffness, rear axle side stiffness, moment of inertia about the vertical axis, distance from the center of mass to the front axle, and distance from the center of mass to the rear axle. Based on the vehicle parameters, construct a state-space expression; The state-space expression is discretized to obtain the first state-space equation.

[0017] As a preferred embodiment, the information acquisition module employs a model predictive control algorithm to optimize the sequence to be optimized, thereby obtaining the control sequence, including: Under preset constraints, the information acquisition module optimizes the sequence to be optimized according to the objective function of the model predictive control algorithm to obtain the control sequence; wherein... The objective function is to minimize the result calculated according to the following cost function: ; Where J is the cost function, Y is the state value sequence matrix in the prediction time domain, and Y ref Let U be the expected state value sequence matrix in the prediction time domain, and ΔU be the sequence to be optimized. k-1 H is the control sequence matrix from the previous time step, where H is U. k-1 The lower triangular matrix obtained by transforming and solving between △U, As slack variables, Here, represents the weighting coefficients of the slack variables, Q represents the weighting matrix of the output error, R represents the weighting matrix of the control increment, and S represents the weighting matrix of the control quantity.

[0018] Accordingly, this application also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the reinforcement learning-based vehicle control method.

[0019] Accordingly, this application also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the reinforcement learning-based vehicle control method.

[0020] Compared with the prior art, this invention application has the following beneficial effects: This invention application provides a vehicle control method, system, device, and medium based on reinforcement learning. The method includes: acquiring vehicle state information and reference trajectory information of a target vehicle at the current moment; wherein, the vehicle state information at the current moment includes the current vehicle position and the current heading angle; based on the vehicle state information and the reference trajectory information, performing rolling optimization solution through a model predictive control algorithm to obtain a control sequence covering the control time domain corresponding to the current moment; wherein, the control sequence is a front wheel steering angle increment sequence; constructing a reinforcement learning state vector based on the vehicle state information, reference trajectory information, and the control sequence and inputting it into a policy network of a pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction amount at the current moment; superimposing the first term of the front wheel steering angle increment, the front wheel steering angle correction amount, and the front wheel steering angle control amount of the previous moment in the control sequence to generate the front wheel steering angle control amount at the current moment. This invention employs an architecture that integrates model predictive control (MMC) algorithm with a flexible motion-evaluation network (FERN). This architecture combines the forward-looking optimization capability of MMC with the data-driven nonlinear mapping capability of FERN. The control sequence provided by MMC, covering the future control time domain, provides accurate prior control trend information to the policy network of FERN, enabling FERN to accurately compensate for errors caused by nonlinearity and uncertainty. This fusion mechanism allows the controller to no longer rely entirely on a fixed vehicle dynamics model, thus effectively overcoming the technical problem of performance degradation or even instability caused by inaccurate models in existing MMC algorithms. It can still maintain high-precision trajectory tracking under complex and variable real driving conditions. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating an embodiment of the vehicle control method based on reinforcement learning provided in this invention application.

[0022] Figure 2 This is a schematic diagram of the technical architecture of an embodiment of the vehicle control method based on reinforcement learning provided in this invention application.

[0023] Figure 3 This is a flowchart illustrating a preferred embodiment of the reinforcement learning-based vehicle control method provided in this invention application.

[0024] Figure 4 This is a flowchart illustrating another preferred embodiment of the reinforcement learning-based vehicle control method provided in this application.

[0025] Figure 5 This is a schematic diagram of an embodiment of a vehicle control system based on reinforcement learning provided in this invention application. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Example 1 Please refer to Figure 1 , Figure 1 The present invention provides a vehicle control method based on reinforcement learning, comprising steps S101 to S104; wherein each step is described in detail below: Step S101: Obtain the current vehicle status information and reference trajectory information of the target vehicle.

[0028] In this step, the real-time status data includes, but is not limited to, the current vehicle position, transmission ratio, steering wheel angle, current longitudinal speed, and current heading angle.

[0029] For example, such as Figure 2 As shown, real-time status data can be acquired through hardware such as the vehicle chassis, body, and positioning system. The current (moment) longitudinal velocity can be represented as v. x (Unit: meters per second), the current heading angle can be expressed as: (Unit: rad) The steering wheel angle can be expressed as... (Unit: rad) The transmission ratio can be expressed as K s .

[0030] In some preferred embodiments, the path planner in the domain controller can obtain reference path information ahead of the nearest point of the vehicle on the current target trajectory within the prediction time domain. The reference trajectory information includes the vehicle's reference position and reference heading angle at the prediction data point closest to the target vehicle on the reference trajectory.

[0031] Step S102: Based on the vehicle state information and reference trajectory information, a rolling optimization solution is performed using a model predictive control algorithm to obtain the control sequence corresponding to the current time in the coverage control time domain.

[0032] In this step, the control sequence is the front wheel steering angle increment sequence.

[0033] In this embodiment, Model Predictive Control (MPC) is used. In the field of control engineering, MPC is widely applied to the control of various complex systems due to its unique advantages. Its core idea is to predict the future behavior of the system based on a model and determine the control input at the current moment by solving a finite-time optimization problem.

[0034] In some preferred embodiments, it is assumed that the target vehicle corresponds to a discrete-time system, for example, described by the following formula: ; Where x(k) is the state variable at the current time, A is the state matrix, and B is the input matrix. This is the state variable for the next moment.

[0035] More preferably, such as Figure 3 As shown, step S102 includes steps S301 to S303; each step is described in detail below: Step S301: Calculate the lateral deviation and lateral deviation rate of change of the target vehicle based on the vehicle reference position and the current vehicle position; calculate the heading deviation and heading deviation rate of change of the target vehicle based on the reference heading angle and the current heading angle. Step S302: Substitute the lateral deviation, lateral deviation rate of change, heading deviation, and heading deviation rate of change into the preset first state space equation corresponding to the target vehicle and perform rolling solution to obtain the sequence to be optimized. Step S303: The model predictive control algorithm is used to optimize the sequence to be optimized, thereby obtaining the control sequence.

[0036] For example, the prediction time domain can be represented as N p The control time domain can be represented as N c The length of the time domain can be represented by the number of time points. The control time domain can be dynamically adjusted based on vehicle speed and the required aiming distance. Assuming the maximum aiming distance is S2 (in meters), the minimum aiming distance is S1 (in meters), and the maximum control speed of the vehicle in longitudinal control is V... max (Unit: meters per second), minimum control speed is V min (Default value is zero, unit is meters per second), which can calculate the distance that needs to be pre-aimed at the current moment. This can then be combined with the current vehicle speed v x Determine the above control time domain N c .

[0037] The predicted data points in the reference trajectory (predicted trajectory) can be represented as A1, A2, A3, ..., An. The predicted data point closest to the target vehicle is A1, and its position can be represented as (x1, y1). The current vehicle position can be represented as (x, y), and the lateral deviation between them is denoted as e. y The aforementioned rate of change of lateral deviation can be expressed as: Predict the heading angle of data point A1 With respect to the vehicle's current heading angle The heading deviation between them can be expressed as (Unit: rad) The rate of change of the above heading deviation is expressed as: (Unit: rad / s).

[0038] In this preferred embodiment, the lateral deviation, the rate of change of lateral deviation, the heading deviation, and the rate of change of heading deviation can be used as state variables. Front wheel steering angle increment u mpc As a control variable, it is substituted into the first state-space equation.

[0039] In some preferred embodiments, the first state-space equation described above can be expressed as: ; In this embodiment, A1, B1, and B2 are intermediate variable matrices, mainly used to simplify calculations; Let be the desired rate of change of heading angle.

[0040] By implementing this preferred embodiment, the predicted data points closest to the target vehicle in the reference trajectory are extracted, and the lateral deviation, lateral deviation rate of change, heading deviation, and heading deviation rate of change are accurately calculated. These parameters are then substituted into the first state-space equation to obtain the sequence to be optimized. The control sequence is then obtained through MPC optimization. This allows the input deviation data to better reflect the difference between the actual vehicle state and the reference trajectory when the MPC solves the control sequence, avoiding deviations in the optimization direction caused by fuzzy deviation calculations. This makes the control sequence output by the MPC more targeted and accurate.

[0041] Preferably, before step S302, the method includes: acquiring vehicle parameters of the target vehicle; wherein the vehicle parameters include vehicle curb weight, front axle side stiffness, rear axle side stiffness, moment of inertia about the vertical axis, distance from the center of mass to the front axle, and distance from the center of mass to the rear axle; constructing a state-space expression based on the vehicle parameters; and discretizing the state-space expression to obtain the first state-space equation.

[0042] For example, the state-space expression is as follows: Where m is the normal curb weight of the target vehicle (in kilograms). The front axle side plate stiffness of the target vehicle (in Newtons / rad). The rear axle side stiffness of the target vehicle (in Newtons / rad). The moment of inertia of the target vehicle about the vertical axis (z-axis) is expressed in kilograms per square meter. The distance from the center of gravity of the target vehicle to the front axle (in meters). The distance from the center of gravity of the target vehicle to the rear axle (in meters). The front wheel steering angle of the target vehicle (in rad).

[0043] Furthermore, based on the state variable x(k), the control variable u(k-1) from the previous time step is added as a state variable to obtain a new state variable (considering the control variable from the previous time step). : ; also , If the control increment is within a unit of time, then a new discrete equation exists: ; ; ; ; ; ; Among them, A, B, A d B d and Let E be the intermediate variable matrix, and T be the identity matrix. s To control the control cycle, I n It is an n x l matrix where the value is 1, 0 n It is an n-row, l-column matrix with all values ​​being 0.

[0044] The first state-space equation obtained through discretization in this embodiment describes the behavior of a continuous-time dynamic system at discrete time points (or a discrete-time system). After obtaining the first state-space equation through discretization, this embodiment can derive the time domain from the current time to the prediction time N step by step according to the time. p The state equations for each step at time k ultimately yield the equations from the current time k up to the Nth time in the prediction time domain. p The prediction matrix at each time step. The prediction matrix Y is the state quantity predicted at each time step in the time domain. Furthermore, because we need the state quantity Y at each time step and the expected state quantity Y0 at each time step...ref The difference reaches its minimum, i.e., the expected value. The value of is 0, which can then be incorporated into the final objective equation for solving.

[0045] Further, step S304 involves using a model predictive control algorithm to optimize the sequence to obtain the control sequence, including: Under preset constraints, the sequence to be optimized is obtained by optimizing the objective function of the model predictive control algorithm to obtain the control sequence; wherein... The objective function is to minimize the result calculated according to the following cost function: ; Where J is the cost function, Y is the state value sequence matrix in the prediction time domain, and Y ref Let U be the expected state value sequence matrix in the prediction time domain, and ΔU be the first sequence to be optimized. k-1 H is the control sequence matrix from the previous time step, where H is U. k-1 The lower triangular matrix obtained by transforming and solving between △U, As slack variables, Here, represents the weighting coefficients of the slack variables, Q represents the weighting matrix of the output error, R represents the weighting matrix of the control increment, and S represents the weighting matrix of the control quantity.

[0046] Preferably, the preset constraints may include front wheel steering angle constraints (such as...) , and These are the preset minimum and maximum front wheel steering angles, and the front wheel steering angle increment constraint (e.g.) , and These are the preset minimum and maximum values ​​of the front wheel steering angle increment, and control constraints (such as...). , and These are the preset minimum and maximum control increment values ​​and control increment constraints (e.g., ...). , and These are the preset minimum and maximum control increment values, respectively.

[0047] Furthermore, the following inequality relationship can be obtained: ; ; Finally, the control sequence in the control time domain is obtained by solving a quadratic programming QP algorithm. (The corresponding image is) ).in Indicates the Nth c The front wheel steering angle increment at time -1.

[0048] Furthermore, it should be noted that the control time domain N c In the model predictive control algorithm optimization problem, this is the number of future control actions that need to be calculated and optimized. This number is determined by N in the control time domain. c Decide.

[0049] Step S103: Construct a reinforcement learning state vector based on the vehicle state information, reference trajectory information, and control sequence; and input the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction amount at the current moment.

[0050] In this step, the full name of the Soft Actor-Critic network is Soft Actor-Critic, abbreviated as SAC network, which includes a policy network, a target Q network, and a master Q network.

[0051] The reinforcement learning state vector constructed based on vehicle state information, reference trajectory information, and control sequence can be represented as: ; Among them, s t Let t be the state vector at time t (the current time). This represents the lateral deviation at time t (in meters). This represents the heading deviation at time t (in rad). This represents the rate of change of the lateral deviation at time t (in meters per second). For prediction of the time domain The curvature of each point within, To predict time domain N p Lateral deviation (in meters) between the vehicle's trajectory and the reference trajectory; To predict time domain N p The heading deviation (in rad) between the vehicle's trajectory and the reference trajectory.

[0052] By setting the state vectors above, the flexible motion-evaluation network not only knows the current vehicle state deviation and judges whether the steering increment given by the model predictive control algorithm is sufficient, excessive or insufficient; it also knows the curvature of the curve ahead and the deviation between the predicted vehicle trajectory and the reference trajectory, thus obtaining strong forward predictive ability and making necessary compensation in advance. To ensure the flexibility of the motion-evaluation network, normalization preprocessing (zero mean, unit variance) can be performed on each component. In some preferred embodiments, the action of the flexible action-evaluation network can be defined as follows: , This refers to the front wheel steering angle correction (in rad) output by the flexible motion evaluation network. It can be constrained as follows: , This represents the limit for the front wheel steering angle correction. This prevents the flexible action evaluation network from making overly aggressive and dangerous movements, maintains the stability of the model predictive control algorithm, and does not overwrite the control characteristics of the model predictive control algorithm.

[0053] In some preferred embodiments, the definition of the reward function for the flexible action-evaluation network may include: Tracking accuracy penalty : ; in, and These are the weighting coefficients for lateral deviation and heading deviation, respectively. , ∈[1,10]; Tracking accuracy stability penalty : ; in, and These are the weighting coefficients for the rate of change of lateral deviation and the rate of change of heading deviation, respectively. Flexible Action - Evaluation Network Algorithm Control Variable Extreme Value Penalty ; ; in, It is the control quantity weighting coefficient. ; Flexible Actions - Evaluation of Network Algorithm Control Variable Stability Penalty: in, It is the stability weighting coefficient of the control quantity. , This represents the front wheel steering angle correction amount output by the network before the last training iteration, representing a flexible motion.

[0054] Based on the definition of the penalty function above, the reward function r t for: ; In some preferred embodiments, such as Figure 4 As shown, step S103, which involves inputting the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction at the current moment, includes steps S401 to S404; each step is detailed below: Step S401: Input the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network.

[0055] Step S402: Based on the output of the policy network, obtain the corresponding Gaussian distribution.

[0056] Step S403: The Gaussian distribution is compressed using the hyperbolic tangent function to obtain the action correction base value.

[0057] Step S404: The action correction base value is corrected by using the preset maximum correction range of the front wheel angle to obtain the front wheel angle correction amount at the current moment.

[0058] For example, the Gaussian distribution described in step S402 : ; in, , Let be a random noise vector sampled from a standard normal distribution with a mean of 0 and a variance of 1. Represents the reinforcement learning state vector s t The corresponding mean, Represents the reinforcement learning state vector s t The corresponding standard deviation.

[0059] Furthermore, step S403 compresses the material using the hyperbolic tangent function, thereby reducing the density of the material. from Mapped to Obtain action correction base value : ; And introduce the maximum correction magnitude Final SAC motion correction amount (The front wheel steering angle correction amount mentioned in step S404) is: .

[0060] In some preferred embodiments, before inputting the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network, the training method of the flexible action-evaluation network includes: A preset experience replay pool is sampled to obtain a certain amount of training data; each piece of training data includes a state vector at a historical moment; the training data is used to train the basic network model to be trained; the basic network model includes a target Q-network and a main Q-network; during each training, the soft Bellman target value for the next moment of the historical moment is calculated through the target Q-network; the soft Bellman target value is used as a supervision signal to update the parameters of the main Q-network by minimizing the mean square error loss, and the parameters of the policy network are updated through the output of the updated main Q-network; the parameters of the target Q-network are updated synchronously in a soft update manner to make it approximate the main Q-network; when the preset convergence condition is met, the trained flexible action-evaluation network is obtained.

[0061] In this embodiment, the experience pool can be represented as D, from which a batch of data can be randomly sampled for training.

[0062] Preferably, the soft Bellman objective value for the next time step of the historical moment is calculated using the target Q-network as follows: ; Where y represents the soft Bellman objective value of the next time step in the historical time. It is based on the current policy network in New actions obtained from downsampling; This is the policy entropy for the next moment.

[0063] To address the Q-value overestimation problem, the outputs of the two target Q-networks are calculated and their minimum value is taken: By comparing the first target action value network (Target Critic 1, parameters are...) ) and the second target action value network (Target Critic 2, parameters are The smaller of the two values ​​is selected as the conservative target value estimate for the state-action pair at the next time step, i.e.: ; Then, the two principal values ​​are updated by minimizing the mean squared error (MSE). Network parameters and : ; Where E represents minimizing the mean squared error function, and i represents the main Q-network number.

[0064] Preferably, step S403, which involves compressing the Gaussian distribution using the hyperbolic tangent function to obtain the action correction base value, specifically involves: The Gaussian distribution is compressed according to the following formula to obtain the action correction base value: ; in, The action correction value at time t (i.e., the reparameterized sampled action, i.e., the final input to) The network performs evaluations or outputs action instructions to the environment for execution. Let s be a random noise vector sampled from a standard normal distribution. t Let be the reinforcement learning state vector. Let be the mean of the Gaussian distribution, representing the central action that the policy considers "most appropriate" to take in the current state. The standard deviation of the Gaussian distribution represents the "uncertainty" or "range of exploration" of the policy with respect to the current action. The larger the size, the greater the fluctuation in movement, and the stronger the exploratory nature. The smaller the value, the closer the movement is to the mean. The more stable the performance, It is the hyperbolic tangent activation function.

[0065] The goal of a policy network is to maximize Maximize entropy while maximizing the expected return. Sample actions using reparameterization techniques. loss function Defined as: in This represents the weight parameters of the policy network. In this step, the policy network adjusts these parameters. The generated action Ratings in the Critic network The value should be as high as possible (i.e., the second term in the formula should minimize the negative value). (value), while keeping the entropy of the action probability distribution as large as possible (i.e., the first term in the formula). (As small as possible, corresponding to high entropy). By comparing the first action value network (Critic 1, parameters are...) ) and the second action value network (Critic 2, parameters are The smaller of the two values ​​is selected as the conservative value estimate for the state-action pair. .

[0066] Preferably, this can also be achieved by minimizing the following temperature coefficient. loss function Automatic temperature coefficient adjustment To ensure that the average entropy of the strategy is close to the given target entropy. : in, Indicates the current policy in state Select action The logarithmic probability, and its opposite This represents the instantaneous entropy of the current strategy at that sample data point; The result after the action represents the average entropy of the strategy; The target entropy, essentially a preset threshold, represents the minimum average information entropy that the policy network should maintain; temperature parameter. loss function Essentially, it calculates the difference between the "average entropy of the current policy" and the "target entropy." This loss function is minimized using gradient descent. The controller can adaptively and dynamically adjust The value.

[0067] The specific adjustment mechanism can be as follows: when the entropy of the current strategy is less than the target entropy... This indicates that the strategy is too deterministic and the exploration is insufficient. In this case, the gradient of the loss function will cause the algorithm to automatically increase the temperature parameter. This increases the weight of entropy in the total reward, forcing the strategy to increase randomness to enhance exploration; when the entropy of the current strategy is greater than the target entropy. When this occurs, it indicates that the strategy is too random and lacks specificity. In this case, the algorithm will automatically reduce the temperature parameter. This reduces the weight of entropy, making the policy more focused on obtaining higher cumulative rewards. Ultimately, this mechanism allows the policy to explore extensively in the early stages of training and converge to the optimal policy in the later stages, effectively preventing the policy from getting trapped in local optima too early.

[0068] Preferably, during training, the target Network parameters ( and Gradient updates can be skipped, and instead, soft updates can be used to follow the master. Network parameters ( , ): ; in, It is a very small smoothing coefficient, which makes the target network parameters change slowly, thereby stabilizing the training objective.

[0069] By repeatedly training the basic network model iteratively until the policy network converges or reaches the preset number of training rounds, a stable flexible action-evaluation network and stable policy network parameters are finally obtained. It is used for real-time online inference of vehicles.

[0070] Step S104: The front wheel angle increment, front wheel angle correction, and front wheel angle control quantity of the first item in the control sequence are superimposed to generate the front wheel angle control quantity of the current moment.

[0071] In this step, based on the above, the first term of front wheel steering angle increment is... Front wheel steering angle correction and the front wheel steering angle control value at the previous moment Add them together to obtain the front wheel steering angle control value at the current moment. for: ; in, The first value (in rad) of the control sequence of the model predictive control algorithm, namely the first term of the front wheel steering angle increment, provides the main control quantity; The compensation output by the flexible motion-evaluation network algorithm is a control increment compensation (unit: rad) for model bias, complex operating conditions, etc. This is the front wheel steering angle control value at the previous moment, which ensures the continuity of control (unit: rad).

[0072] Furthermore, the front wheel steering angle control value can be converted into the steering wheel steering angle control value using the following formula. : ; The steering wheel angle control signal is transmitted to the EPS actuator via the CAN bus to control the target vehicle and obtain the next speed adjustment signal. t+1 and reward r t It is used for control in the next moment.

[0073] Accordingly, such as Figure 5 As shown, this invention application also provides a vehicle control system 500 based on reinforcement learning, including an information acquisition module 501, a solution module 502, a correction module 503, and a generation module 504; wherein, The information acquisition module 501 is used to acquire the vehicle status information and reference trajectory information of the target vehicle at the current moment; wherein, the vehicle status information at the current moment includes the current vehicle position and the current heading angle; The solution module 502 is used to perform rolling optimization based on the vehicle state information and reference trajectory information using a model predictive control algorithm to obtain the control sequence covering the control time domain at the current moment; wherein, the control sequence is the front wheel steering angle increment sequence; The correction module 503 is used to construct a reinforcement learning state vector based on the vehicle state information, reference trajectory information and control sequence, and input the reinforcement learning state vector into the policy network of a pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction amount at the current moment. The generation module 504 is used to superimpose the first item of the control sequence, the front wheel angle increment, the front wheel angle correction, and the front wheel angle control quantity of the previous moment to generate the front wheel angle control quantity of the current moment.

[0074] As a preferred embodiment, the correction module 503 inputs the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction amount at the current moment, including: The correction module 503 inputs the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network; Based on the output of the policy network, the corresponding Gaussian distribution is obtained; The Gaussian distribution is compressed using the hyperbolic tangent function to obtain the action correction base value; The action correction base value is corrected by using the preset maximum correction range of the front wheel angle to obtain the front wheel angle correction amount at the current moment.

[0075] As a preferred embodiment, the vehicle control system 500 further includes a training module, which is used before the correction module 503 inputs the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network: A preset experience replay pool is sampled to obtain a certain number of training data; wherein each piece of training data includes a state vector at a historical moment; Using the training data, a basic network model to be trained is trained; wherein, the basic network model includes a target Q-network and a main Q-network; During each training session, the soft Bellman target value for the next time step of the historical moment is calculated using the target Q-network. The soft Bellman objective value is used as a supervision signal. The parameters of the main Q network are updated by minimizing the mean square error loss. The parameters of the policy network are then updated using the output of the updated main Q network. When the preset convergence condition is met, the trained flexible action-evaluation network is obtained.

[0076] As a preferred embodiment, the correction module 503 compresses the Gaussian distribution using a hyperbolic tangent function to obtain the action correction base value, specifically: The correction module 503 performs compression processing on the Gaussian distribution according to the following formula to obtain the action correction base value: ; in, The action correction value at time t. Let s be a random noise vector sampled from a standard normal distribution. t Let be the reinforcement learning state vector. The mean of the Gaussian distribution is given. Let be the standard deviation of the Gaussian distribution.

[0077] As a preferred embodiment, the reference trajectory information includes the vehicle reference position and reference heading angle of the predicted data point closest to the target vehicle in the reference trajectory; The information acquisition module 501, based on the vehicle state information and reference trajectory information, performs rolling optimization using a model predictive control algorithm to obtain the control sequence covering the control time domain corresponding to the current moment, including: The information acquisition module 501 calculates the lateral deviation and lateral deviation rate of change of the target vehicle based on the vehicle reference position and the current vehicle position; and calculates the heading deviation and heading deviation rate of change of the target vehicle based on the reference heading angle and the current heading angle. Substitute the lateral deviation, lateral deviation rate of change, heading deviation, and heading deviation rate of change into the preset first state space equation corresponding to the target vehicle and perform rolling solution to obtain the sequence to be optimized. The control sequence is obtained by using a model predictive control algorithm to optimize the sequence to be optimized.

[0078] As a preferred embodiment, the vehicle control system 500 further includes an equation construction module, which is used before the information acquisition module 501 substitutes the lateral deviation, lateral deviation rate of change, heading deviation, and heading deviation rate of change into the preset first state space equation corresponding to the target vehicle for rolling solution: Obtain the vehicle parameters of the target vehicle; wherein the vehicle parameters include the vehicle's curb weight, front axle side stiffness, rear axle side stiffness, moment of inertia about the vertical axis, distance from the center of mass to the front axle, and distance from the center of mass to the rear axle. Based on the vehicle parameters, construct a state-space expression; The state-space expression is discretized to obtain the first state-space equation.

[0079] As a preferred embodiment, the information acquisition module 501 employs a model predictive control algorithm to optimize the sequence to be optimized, thereby obtaining the control sequence, including: Under preset constraints, the information acquisition module 501 optimizes the sequence to be optimized according to the objective function of the model predictive control algorithm to obtain the control sequence; wherein... The objective function is to minimize the result calculated according to the following cost function: ; Where J is the cost function, Y is the state value sequence matrix in the prediction time domain, and Y ref Let U be the expected state value sequence matrix in the prediction time domain, and ΔU be the sequence to be optimized. k-1 H is the control sequence matrix from the previous time step, where H is U. k-1 The lower triangular matrix obtained by transforming and solving between △U, As slack variables, Here, represents the weighting coefficients of the slack variables, Q represents the weighting matrix of the output error, R represents the weighting matrix of the control increment, and S represents the weighting matrix of the control quantity.

[0080] Accordingly, this application also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the reinforcement learning-based vehicle control method.

[0081] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal, connecting various parts of the terminal via various interfaces and lines.

[0082] The memory can be used to store the computer program. The processor implements various functions of the terminal by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0083] Accordingly, this application also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the reinforcement learning-based vehicle control method.

[0084] Wherein, if the modules of the device / terminal equipment / system integration are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0085] Compared with the prior art, this invention application has the following beneficial effects: This invention application provides a vehicle control method, system, device, and medium based on reinforcement learning. The method includes: acquiring vehicle state information and reference trajectory information of a target vehicle at the current moment; wherein, the vehicle state information at the current moment includes the current vehicle position and the current heading angle; based on the vehicle state information and the reference trajectory information, performing rolling optimization solution through a model predictive control algorithm to obtain a control sequence covering the control time domain corresponding to the current moment; wherein, the control sequence is a front wheel steering angle increment sequence; constructing a reinforcement learning state vector based on the vehicle state information, reference trajectory information, and the control sequence and inputting it into a policy network of a pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction amount at the current moment; superimposing the first term of the front wheel steering angle increment, the front wheel steering angle correction amount, and the front wheel steering angle control amount of the previous moment in the control sequence to generate the front wheel steering angle control amount at the current moment. This invention employs an architecture that integrates model predictive control (MMC) algorithm with a flexible motion-evaluation network (FERN). This architecture combines the forward-looking optimization capability of MMC with the data-driven nonlinear mapping capability of FERN. The control sequence provided by MMC, covering the future control time domain, provides accurate prior control trend information to the policy network of FERN, enabling FERN to accurately compensate for errors caused by nonlinearity and uncertainty. This fusion mechanism allows the controller to no longer rely entirely on a fixed vehicle dynamics model, thus effectively overcoming the technical problem of performance degradation or even instability caused by inaccurate models in existing MMC algorithms. It can still maintain high-precision trajectory tracking under complex and variable real driving conditions.

[0086] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A vehicle control method based on reinforcement learning, characterized by, include: Obtain the current vehicle status information and reference trajectory information of the target vehicle; wherein, the current vehicle status information includes the current vehicle position and the current heading angle; Based on the vehicle state information and reference trajectory information, a rolling optimization solution is performed using a model predictive control algorithm to obtain the control sequence covering the control time domain at the current moment; wherein, the control sequence is the front wheel steering angle increment sequence; A reinforcement learning state vector is constructed based on the vehicle state information, reference trajectory information, and control sequence; and the reinforcement learning state vector is input into the policy network of a pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction at the current moment. The front wheel angle increment, front wheel angle correction, and front wheel angle control quantity from the previous moment are superimposed in the control sequence to generate the front wheel angle control quantity at the current moment.

2. The vehicle control method based on reinforcement learning as described in claim 1, characterized in that, The step of inputting the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction at the current moment includes: The reinforcement learning state vector is input into the policy network of a pre-trained flexible action-evaluation network; Based on the output of the policy network, the corresponding Gaussian distribution is obtained; The Gaussian distribution is compressed using the hyperbolic tangent function to obtain the action correction base value; The action correction base value is corrected by using the preset maximum correction range of the front wheel angle to obtain the front wheel angle correction amount at the current moment.

3. The vehicle control method based on reinforcement learning as described in claim 2, characterized in that, Before inputting the reinforcement learning state vector into the policy network of the pre-trained flexible action-evaluation network, the following steps are included: A preset experience replay pool is sampled to obtain a certain number of training data; wherein each piece of training data includes a state vector at a historical moment; Using the training data, a basic network model to be trained is trained; wherein, the basic network model includes a target Q-network and a main Q-network; During each training session, the soft Bellman target value for the next time step of the historical moment is calculated using the target Q-network. The soft Bellman objective value is used as a supervision signal. The parameters of the main Q network are updated by minimizing the mean square error loss. The parameters of the policy network are then updated using the output of the updated main Q network. When the preset convergence condition is met, the trained flexible action-evaluation network is obtained.

4. The vehicle control method based on reinforcement learning as described in claim 2, characterized in that, The step of compressing the Gaussian distribution using the hyperbolic tangent function to obtain the action correction base value is as follows: The Gaussian distribution is compressed according to the following formula to obtain the action correction base value: ； in, The action correction value at time t. Let s be a random noise vector sampled from a standard normal distribution. t Let be the reinforcement learning state vector. The mean of the Gaussian distribution is given. Let be the standard deviation of the Gaussian distribution.

5. The vehicle control method based on reinforcement learning as described in claim 1, characterized in that, The reference trajectory information includes the vehicle reference position and reference heading angle of the predicted data point closest to the target vehicle in the reference trajectory; Based on the vehicle state information and reference trajectory information, a rolling optimization solution is performed using a model predictive control algorithm to obtain the control sequence corresponding to the current time-domain coverage control, including: Based on the vehicle reference position and the current vehicle position, calculate the lateral deviation and lateral deviation rate of change of the target vehicle; based on the reference heading angle and the current heading angle, calculate the heading deviation and heading deviation rate of change of the target vehicle. Substitute the lateral deviation, lateral deviation rate of change, heading deviation, and heading deviation rate of change into the preset first state space equation corresponding to the target vehicle and perform rolling solution to obtain the sequence to be optimized. The control sequence is obtained by using a model predictive control algorithm to optimize the sequence to be optimized.

6. The vehicle control method based on reinforcement learning as described in claim 5, characterized in that, Before substituting the lateral deviation, lateral deviation rate of change, heading deviation, and heading deviation rate of change into the preset first state space equation corresponding to the target vehicle for rolling solution, the method further includes: Obtain the vehicle parameters of the target vehicle; wherein the vehicle parameters include the vehicle's curb weight, front axle side stiffness, rear axle side stiffness, moment of inertia about the vertical axis, distance from the center of mass to the front axle, and distance from the center of mass to the rear axle. Based on the vehicle parameters, construct a state-space expression; The state-space expression is discretized to obtain the first state-space equation.

7. The vehicle control method based on reinforcement learning as described in claim 5, characterized in that, The model predictive control algorithm is used to optimize the sequence to be optimized, resulting in the control sequence, including: Under preset constraints, the sequence to be optimized is obtained by optimizing the objective function of the model predictive control algorithm to obtain the control sequence; wherein... The objective function is to minimize the result calculated according to the following cost function: ； Where J is the cost function, Y is the state value sequence matrix in the prediction time domain, and Y ref Let U be the expected state value sequence matrix in the prediction time domain, and ΔU be the sequence to be optimized. k-1 H is the control sequence matrix from the previous time step, where H is U. k-1 The lower triangular matrix obtained by transforming and solving between △U, As slack variables, Here, represents the weighting coefficients of the slack variables, Q represents the weighting matrix of the output error, R represents the weighting matrix of the control increment, and S represents the weighting matrix of the control quantity.

8. A vehicle control system based on reinforcement learning, characterized in that, It includes an information acquisition module, a solution module, a correction module, and a generation module; among which, The information acquisition module is used to acquire the vehicle status information and reference trajectory information of the target vehicle at the current moment; wherein, the vehicle status information at the current moment includes the current vehicle position and the current heading angle; The solution module is used to perform rolling optimization based on the vehicle state information and reference trajectory information using a model predictive control algorithm to obtain the control sequence covering the control time domain at the current moment; wherein, the control sequence is the front wheel steering angle increment sequence; The correction module is used to construct a reinforcement learning state vector based on the vehicle state information, reference trajectory information and control sequence, and input the reinforcement learning state vector into the policy network of a pre-trained flexible action-evaluation network to obtain the front wheel steering angle correction amount at the current moment. The generation module is used to superimpose the first front wheel angle increment, the front wheel angle correction, and the front wheel angle control quantity of the previous moment in the control sequence to generate the front wheel angle control quantity of the current moment.

9. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the reinforcement learning-based vehicle control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the reinforcement learning-based vehicle control method as described in any one of claims 1 to 7.

Images