Self-learning motion control method for a multi-axle steered vehicle

Abstract

The application discloses a self-learning motion control method of a multi-axle steering vehicle, and specifically comprises the following steps: self-learning motion control based on a reinforcement learning algorithm, including obtaining self-state information from a vehicle sensor and obtaining a preset trajectory from an automatic driving trajectory planning system; outputting a current ideal multi-axle steering mode according to input state information by using a self-learning intelligent agent trained based on a reinforcement algorithm; and delivering the instruction to a state machine to control whether each steering shaft of the multi-axle vehicle participates in steering. According to the application, the tracking performance of the vehicle during turning is improved, the efficiency of the vehicle during driving is ensured, and a feasible solution is provided for the combination of an automatic driving system and a multi-axle steering motion technology of the vehicle.

Description

Technical Field

[0001] This invention relates to the technical field of multi-axle vehicle motion control, and in particular to a self-learning motion control method based on deep reinforcement learning algorithms. Background Technology

[0002] With the rapid development of China's automotive and transportation industries, the application of large-scale special-purpose transport vehicles in various sectors is becoming increasingly widespread. In transportation, civil engineering, and military transport, there are higher demands and requirements for large and medium-sized vehicles compared to the past. Traditional large vehicles suffer from drawbacks such as large turning radii, poor handling stability, and severe tire wear. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide a self-learning motion control method based on deep reinforcement learning algorithms, combining autonomous driving technology with multi-axis steering technology to achieve trajectory tracking and autonomous driving in multi-axis steering vehicles. To achieve the above-mentioned objectives and other advantages of the present invention, a self-learning motion control method based on deep reinforcement learning algorithms is provided, comprising:

[0004] S1. Obtain real-time status information of the vehicle during driving through onboard sensors and transmit the real-time information to the state machine;

[0005] S2. Establish an initial control model for self-learning motion control of a multi-axis steering vehicle for autonomous driving by using real-time state information and reinforcement learning algorithms, and design the corresponding state space, action space and reward function.

[0006] S3. Train the initial model through simulation scenarios to obtain the final model;

[0007] S4. Control the state transitions of the state machine through the final model;

[0008] The vehicle state space in the initial control model is represented as follows: The components include the lateral offset y of the vehicle's relative trajectory, the vehicle speed v, the yaw angle φ, and the yaw rate. and the curvature k of the forward trajectory;

[0009] The action space is A = {a1, a2, a3}; where a1, a2, and a3 are the control instructions of the state machine, which respectively control whether the three steering axes of the car participate in steering;

[0010] The reward function is:

[0011]

[0012] Where k1, k2, and k3 are the weighting coefficients for each reward; the first item is the lateral deviation reward, y, which represents the lateral deviation of the vehicle from the preset trajectory; the second item is the stability reward. The first term represents the yaw rate of the car; the second term is the driving efficiency bonus, v, which represents the expected speed planned by the autonomous driving system; and the third term is the steering mode bonus, which is related to whether each steering axis participates in steering.

[0013] Preferably, in step S1, the state machine controls the steering mode and steering angle used by the autonomous multi-axle steering vehicle during steering. Specifically, the state machine state transition controls whether each axle of the three-axle vehicle participates in steering. Based on the current vehicle state parameters, the MPC algorithm controls the steering angle of each axle participating in steering. The parameters required for the MPC control algorithm are set according to the vehicle structural parameters.

[0014] Preferably, the MPC algorithm is embedded in a state machine, and the optimization objective of the algorithm is: in It reflects the system's ability to follow a reference trajectory; This reflects the limitation on changes in the control quantity, which makes the vehicle control process smoother; N p With N c ε represents the prediction time domain and control time domain in the MPC problem, respectively; Q and R are the weight matrices for penalizing the system state and the control input, respectively; ε is the relaxation factor; and ρ is the weighting coefficient.

[0015] Preferably, step S3 specifically includes the following steps:

[0016] S31. Initialize the Q network parameters θ and Q′ network parameters θ′ of the reinforcement learning model, set the step size parameters a, ε and the maximum number of training units m, where Q is the current network and Q′ is the target network;

[0017] S32. Calculate the reward value R for performing this action in the current state S according to the reward function, and obtain the next state S′ of the car based on the state information;

[0018] S33. Set up an experience pool to store the transfer data from the previous step {S t A t ,R,S′ t When the number of samples in the experience pool exceeds its capacity, the oldest experience sample will be replaced by a new experience sample, where S t Let A be the state of the car at time t. t For in S t The action to be performed in state S, where R is the action performed in state S. t Next, execute action A t The reward value R obtained;

[0019] S34. Randomly select experience samples from the experience pool to update the online network Q network. After training meets certain conditions, update the Q network based on the current Q network parameters θ and Q′ network parameters θ′.

[0020] S35. Input the state information into the Q network to obtain the Q values ​​corresponding to all actions in the action space under the current state S;

[0021] S36. Based on the calculated Q value, select action A according to the ∈-greedy policy, and transmit action instruction A to the state machine.

[0022] Preferably, the current Q network and the target Q′ network have a structure of one input layer, two fully connected layers and one output layer, and the activation function of the output layer is the Tanh function;

[0023] The network update method uses gradient descent for updating, and the loss function formula is:

[0024] Loss(θ)=E(r+γmaxQ(s′,a′,θ′)-Q(s,a,θ));

[0025] Where γ is the attenuation coefficient, Q(s, a, θ) is the Q value of the current network output, and Q(s′, a′, θ′) is the Q value of the target network output.

[0026] Compared with existing technologies, the advantages of this invention are as follows: by inputting the acquired vehicle position, attitude, and other information into a trained self-learning model, the ideal multi-axis steering mode of the current vehicle is obtained, and the instructions are transmitted to the state machine to control whether the three steering axes of the vehicle participate in steering. This achieves both improved tracking performance during vehicle turning and ensured efficiency during vehicle operation. It provides a feasible solution for combining autonomous driving systems with automotive multi-axis steering technology. Attached Figure Description

[0027] Figure 1 A flowchart illustrating the self-learning motion control method based on deep reinforcement learning algorithm according to the present invention;

[0028] Figure 2 This is a flowchart illustrating the training of the self-learning model in a simulation environment for the self-learning motion control method based on deep reinforcement learning algorithm according to the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] Reference Figure 1 and Figure 2 A self-learning motion control method based on deep reinforcement learning algorithms includes:

[0031] S1: Acquire real-time status information of the vehicle during driving through intelligent vehicle onboard sensors. The information is then transmitted to the control module.

[0032] S2: Design a state machine and use the state machine to control the steering mode and steering angle used by the autonomous multi-axis steering vehicle when steering.

[0033] S3: Based on the information in S1, the initial control model of an autonomous multi-axis steering vehicle can be established using reinforcement learning self-learning algorithms, and the corresponding state space, action space and reward function can be designed.

[0034] S4: Train the initial model from S3 using the simulation scenario to obtain the final model.

[0035] S5: Use the final model described in S4 to control the state transitions of the state machine described in S2.

[0036] In S1, in order to obtain the input state information of the control model, it is necessary to obtain the corresponding state information of the car, including the lateral offset y, the car speed v, the yaw angle φ, and the yaw rate. And the curvature k of the forward trajectory. Where, the vehicle speed v, yaw angle φ, and yaw rate are... Information such as these can be obtained from the vehicle's onboard sensors. Therefore, the autonomous multi-axle steering vehicle used in this invention should be equipped with onboard sensors including positioning devices and attitude sensors. Information such as lateral offset y and the curvature k of the forward trajectory are obtained through the path planning module in the autonomous driving system. Therefore, the autonomous multi-axle steering vehicle used in this invention should have an autonomous driving system that can provide path planning for the control module. This system can provide a preset trajectory for the autonomous vehicle based on the actual usage needs of the vehicle and surrounding environmental information.

[0037] In S2, the state machine controls the steering mode used by the autonomous multi-axis steering vehicle during steering. By receiving commands from the control module, the state machine controls whether each steering axis of the vehicle turns and controls the steering angle of the steering axis. To output an ideal steering angle and achieve better trajectory tracking, an MPC control algorithm is used, combining the vehicle's own position information and structural parameters to obtain corresponding control quantities, controlling each steering axis of the vehicle to turn according to the currently desired driving direction, thereby achieving ideal trajectory tracking.

[0038] In S2, the state machine embeds an MPC control algorithm, which can control the vehicle's steering angle based on the planned trajectory and the vehicle's current position information. The optimization objective of the MPC algorithm is: in It reflects the system's ability to follow a reference trajectory; This reflects the limitation on changes in the control quantity, which makes the vehicle control process smoother; N p With N c Let represent the prediction and control time domains in the MPC problem, respectively; Q and R be the weight matrices for penalizing the system state and the control input, respectively; ε be the relaxation factor; and ρ be the weighting coefficients. Then, the designed MPC controller is parameter-tuned. To ensure the effectiveness of the control input, constraints are established on the control input, u... min ≤u(k+i|k)≤u max The optimal control sequence ΔU for the control increment is obtained by solving the optimization problem in the rolling time domain using an MPC controller. * (k)=[Δu * (k), Au * (k+1), ..., Au * (k+N c -1)] T According to the principle of model predictive control, the first term of the control sequence is taken as the control increment signal of the system, and the control quantity u(k) = u(k-1) + Au is determined by combining the output quantity with the output quantity of the previous time step. * (k).

[0039] In S3, to establish a reinforcement learning model for controlling the steering patterns of a multi-axis steering vehicle, a self-learning algorithm is used to build the model. This self-learning algorithm is a deep reinforcement learning algorithm based on value functions. It uses a neural network to approximate the value functions, transforming the Q-table in Q-learning into a Q-network. The algorithm primarily trains the model through experience playback and a double Q-network structure to help the autonomous driving system make decisions and control whether each steering axis of the vehicle participates in steering. The model takes the vehicle's state information S as input and outputs the Q-values ​​of each action in the action space under that state, thereby determining the output action. The state space S is specifically represented as follows: Since the main purpose of adjusting the multi-axis steering mode is to enable the car to better conform to the planned trajectory during steering, the designed state space includes lateral offset y, vehicle speed v, yaw angle φ, and yaw rate. And the curvature k of the forward trajectory. During the model training process, a reward function is designed that considers the stability, accuracy, and efficiency of the car's driving process.

[0040] In S4, the simulation scenario is implemented using professional autonomous driving simulation software. The software establishes a simulation environment, which mainly includes a large number of curved paths with different curvatures and lengths. The validated car model defines important parameters such as the vehicle's mass and structural characteristics.

[0041] The training model method specifically includes the following steps:

[0042] S401: Initialize the Q-network parameters θ and Q′-network parameters θ′ of the self-learning model, where Q is the current network and Q′ is the target network. The current Q-network and the target Q′ network are two neural networks with the same structure but different parameters. In this invention, the network structure consists of one input layer, two fully connected layers, and one output layer. The output layer activation function is the Tanh function, and the other layers use the ReLU function. Then, the neural network is trained so that Q(s, a) gradually approximates Q. * (s, a).

[0043] S402: Then initialize the required parameters, including setting the step size parameter a, ε, and the maximum number of training units m;

[0044] S403: After the system executes an action, calculate the reward value R for executing the action in the current state S according to the reward function, and obtain the next state S′ of the car based on the state information;

[0045] S404: Set up an experience pool to store the transfer data from the previous step {S t A t ,R,S′ t When the number of samples in the experience pool exceeds its capacity, the oldest experience sample will be replaced by a new one. The experience pool capacity determines how many sets of transition data are stored during training. A well-designed experience pool can affect the algorithm's stability, learning speed, and convergence performance.

[0046] Among them, S t Let A be the state of the car at time t. t For in S t The action to be performed in state S, where R is the action performed in state S. t Next, execute action A t The reward value R obtained;

[0047] S405: Randomly select experience samples from the experience pool to update the online Q network. After training meets certain conditions, update the target Q network parameters θ′ based on the current Q network parameters θ.

[0048] The network parameter θ is updated using gradient descent. The mean squared error loss function for the Q network value is expressed as: Loss(θ) = E(r + γmaxQ(s′,a′,θ′) - Q(s,a,θ));

[0049] Where γ is the attenuation coefficient, Q(s, a, θ) is the Q value of the current network output, and Q(s′, a′, θ′) is the Q value of the target network output.

[0050] S406: Input the state information into the Q network to obtain the Q values ​​corresponding to all actions in the action space under the current state S. In this invention, the preset action space is A = {a1, a2, a3}, where a1, a2, and a3 represent whether the three steering axes participate in steering, respectively, and are represented by binary signals 0 or 1.

[0051] Based on the calculated value Q, action A is selected according to the ∈-greedy policy, and action instruction A is transmitted to the state machine. The ∈-greedy policy selects the currently known optimal action with a probability of 1-∈, and randomly selects an action with a probability of ∈.

[0052] Finally, the well-trained model was applied to an autonomous driving system. By passing the state information as input to the Q-network, we were able to obtain the Q-values ​​for all actions in the action space under the current state S. Based on the calculated Q-values, we used an ∈-greedy policy to select an action A and transmitted this action instruction to the state machine, thereby enabling the agent to make decisions and take actions in the environment. In this way, the use of a greedy policy makes the agent more likely to take actions with higher cumulative rewards, thus gradually optimizing its decision-making strategy during the learning process.

[0053] The number of devices and processing scale described herein are for the purpose of simplifying the description of the invention, and applications, modifications and variations thereof will be apparent to those skilled in the art.

[0054] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A self-learning motion control method for a multi-axle steering vehicle, characterized in that, Includes the following steps: S1. Acquire real-time status information of the vehicle during driving through onboard sensors and transmit the real-time status information to the state machine; S2. Establish an initial control model for self-learning motion control of a multi-axis steering vehicle for autonomous driving by using real-time state information and reinforcement learning algorithms, and design the corresponding state space, action space and reward function. S3. Train the initial model through simulation scenarios to obtain the final model; S4. Control the state transitions of the state machine through the final model; The vehicle state space in the initial control model is represented as follows: The lateral deviation of the vehicle from the preset trajectory Expected speed of autonomous driving system planning lateral angle yaw rate curvature of the forward trajectory ; Action space is ;in , and These are the control commands for the state machine, which control whether the three steering axes of the car participate in steering; The reward function is: ； in, , , These are the weighting coefficients for each reward; the first item is the lateral deviation reward. The first item represents the lateral deviation of the vehicle from the preset trajectory; the second item is the stability bonus. The first item represents the car's yaw rate; the third item is the driving efficiency bonus. The first item represents the expected speed planned by the autonomous driving system; the fourth item is the steering mode reward, which is related to whether each steering axis participates in steering.

2. The self-learning motion control method for a multi-axle steering vehicle as described in claim 1, characterized in that, In step S1, the state machine controls the steering mode and steering angle used by the autonomous multi-axle steering vehicle when steering. Specifically, the state machine state transition controls whether each axle of the three-axle vehicle participates in steering. Based on the current state parameters of the vehicle, the MPC algorithm controls the steering angle of each axle participating in steering.

3. The self-learning motion control method for a multi-axle steering vehicle as described in claim 2, characterized in that, The MPC algorithm is embedded in a state machine, and its optimization objective is: ,in It reflects the system's ability to follow a reference trajectory; This reflects the limitation on the changes in control quantity, which makes the vehicle control process smoother; and These represent the prediction time domain and control time domain in the MPC problem, respectively. and These are the weight matrices for penalizing the system state and controlling the input variables, respectively. It is a relaxation factor; These are the weighting coefficients.

4. The self-learning motion control method for a multi-axle steering vehicle as described in claim 3, characterized in that, Step S3 specifically includes the following steps: S31. Initialize the reinforcement learning model Network parameters and Network parameters Set step size parameters , and maximum number of training samples ,in For the current network, For the target network; S32. Calculate the current state based on the reward function. The reward value for performing this action And obtain the car's next state based on the state information. ; S33. Set up an experience pool to store the transfer data from the previous step. When the number of samples in the experience pool exceeds its capacity, the oldest experience sample will be replaced by a new experience sample. Let t be the state of the car. In order to be in Actions performed in a state In the state Next action The reward value obtained ; S34. Randomly select experience samples from the experience pool and analyze them on the online network. The network is updated after training meets certain conditions, based on the current... Network parameters and Network parameters ; S35. Input the status information into the... Get the current status from the network All actions corresponding to the lower action space value; S36. Based on the calculation results The value is used to select action A based on the ϵ-greedy strategy, and the action instruction A is transmitted to the state machine.

5. The self-learning motion control method for a multi-axle steering vehicle as described in claim 4, characterized in that, Among them, the current Network and Target The network structure consists of one input layer, two fully connected layers, and one output layer, with the Tanh function serving as the activation function for the output layer. The network update method uses gradient descent for updating, and the loss function formula is: ; in The attenuation coefficient is... For the current network output value, Output for the target network value.

Images