An optimal control method for trajectory tracking of an omnidirectional car based on reinforcement learning

Abstract

The present invention belongs to the technical field of reinforcement learning and optimal control, and specifically discloses an optimal control method for trajectory tracking of an omnidirectional car based on reinforcement learning, which includes the following steps: establishing a trajectory kinematic model and a dynamic model of the omnidirectional car; designing an optimal control algorithm for trajectory tracking of the omnidirectional car based on a neural network solution based on the trajectory kinematic model and the dynamic model of the omnidirectional car; analyzing the consistent ultimate bounded stability of the omnidirectional car system based on the trajectory tracking optimal control algorithm and continuous excitation conditions of the omnidirectional car, and completing the optimal control method for trajectory tracking of the omnidirectional car based on reinforcement learning. The present invention solves the problem of traditional control algorithms being unable to meet the requirements of traditional control algorithms. HJB The difficult problem of solving the equations is solved, and the condition restriction that the initial control strategy must be admissible control is removed, thus achieving optimal control of the trajectory tracking of the omnidirectional car.

Description

Technical Field

[0001] The invention belongs to the technical field of reinforcement learning and optimal control, and specifically relates to the design of an optimal control method for omnidirectional vehicle trajectory tracking based on reinforcement learning. Background Art

[0002] Omnidirectional cars are widely used in industrial automation, logistics, transportation, service robots and other fields due to their flexible movement and efficient maneuverability. With the continuous development of automation and artificial intelligence technology, simply pursuing the improvement of control performance can no longer meet the needs of practical applications. How to solve the optimization problem of control performance and input energy consumption has attracted more and more attention and research from relevant experts and scholars.

[0003] However, traditional optimal control algorithms often face difficulties when dealing with nonlinear systems such as omnidirectional vehicles. HJB (Hamilton-Jacobi-Bellman) equations are difficult to solve. This is because HJB The nonlinear and coupled characteristics of the equations make analytical solutions almost impossible. In addition, when designing the initial control strategy, traditional optimal control algorithms usually need to ensure the admissibility of the strategy, which further increases the complexity of designing the optimal controller for omnidirectional vehicle trajectory tracking.

[0004] At present, the mainstream optimal control methods for omnidirectional car trajectory tracking include adaptive dynamic programming, fuzzy control and reinforcement learning. The above algorithms all have different parameter update law design methods, but the design of most update laws has certain limitations. The restrictions on the initial control strategy and the processing of strict feedback systems will lead to the complexity of algorithm implementation and the inability to achieve a better trade-off between control performance and input energy consumption. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the traditional optimal control method is difficult to solve and the initial control strategy must be subject to the conditional restrictions of allowable control. An omnidirectional car trajectory tracking optimal control method based on reinforcement learning is proposed.

[0006] The technical solution of the present invention is: an optimal control method for omnidirectional vehicle trajectory tracking based on reinforcement learning, comprising the following steps:

[0007] S1. Establish the trajectory kinematics model and dynamics model of the omnidirectional car;

[0008] S2. Based on the trajectory kinematics model and dynamics model of the omnidirectional car, design the optimal control algorithm for the trajectory tracking of the omnidirectional car based on neural network solution;

[0009] S3. Based on the trajectory tracking optimal control algorithm and continuous excitation conditions of the omnidirectional car, the uniformly ultimately bounded stability of the omnidirectional car system is analyzed, and the optimal control of the trajectory tracking of the omnidirectional car based on reinforcement learning is completed.

[0010] The beneficial effects of the present invention are:

[0011] By introducing a neural network to approximate the optimal value function and the optimal control strategy, and designing a network weight update law with a stability term, the optimal control of the trajectory tracking of the omnidirectional vehicle is achieved, which solves the problem that the traditional optimal control method is difficult to solve, and removes the condition that the initial control strategy must be an admissible control.

[0012] Preferably, the step S1 specifically includes the following formula:

[0013] An omnidirectional car with three DC motors as power devices is selected as the controlled object. The three DC motors respectively drive three Mecanum wheels evenly distributed at intervals of 120°. The omnidirectional motion of the car is achieved through the differential rotation of the wheels. The trajectory kinematic model and dynamic model of the omnidirectional car are modeled as follows:

[0014]

[0015] in, Indicates the position of the omnidirectional car , and azimuth The derivative of a three-dimensional column vector, Indicates the linear speed of the omnidirectional car , and angular velocity The three-dimensional column vector of Represents a three-dimensional column vector The derivative of Indicated by azimuth is the system matrix of parameters, , express dimensional constant matrix, , Indicates the mass of the car, represents the moment of inertia of the car, express dimensional state matrix, , represents the angular velocity, express dimensional constant matrix, , represents the average distance from the geometric center of the car to the wheel, represents the system control input three-dimensional column vector, Indicates the motor torque constant;

[0016] The trajectory kinematics model and dynamics model of the omnidirectional car are organized to obtain a strict feedback form:

[0017]

[0018] in, Indicates the position of the omnidirectional car , and azimuth A three-dimensional column vector, Indicates the linear speed of the omnidirectional car , and angular velocity The three-dimensional column vector of represents the control input, , , , and Both represent system matrices, , , , , Indicates the wheel radius.

[0019] Preferably, the step S2 specifically includes the following sub-steps:

[0020] S21. Based on the trajectory kinematic model and dynamic model of the omnidirectional car, a reference trajectory to be tracked by the omnidirectional car is set, and a tracking error is introduced and the tracking error dynamics is calculated;

[0021] S22. Dynamically define the omnidirectional car tracking performance index based on the tracking error, and obtain the Hamiltonian equation according to the value function definition;

[0022] S23. According to Hamilton's equation, we get HJB Equation;

[0023] S24. Introducing neural network HJB The optimal value function in the equation is approximated, and according to HJB The analytical relationship of the equation is used to update the control strategy;

[0024] S25. Definition HJB The residual term of the equation is used to design the neural network weight update law based on the residual normalized gradient descent method;

[0025] S26. Introduce a stabilization term into the neural network weight update law, obtain the neural network weight update law with the stabilization term, and complete the design of the optimal control algorithm for omnidirectional car trajectory tracking based on reinforcement learning.

[0026] Preferably, the expression formula of the reference trajectory to be tracked by the omnidirectional vehicle in step S21 is:

[0027]

[0028] in, Indicates the reference position of the omnidirectional car , and the reference azimuth A three-dimensional column vector, Indicates the reference linear speed of the omnidirectional car , and the reference angular velocity A three-dimensional column vector, and Respectively and The derivative of , and All represent system matrices;

[0029] The tracking error Including tracking error and tracking error , and its calculation formula is:

[0030]

[0031] in, Indicates the position of the omnidirectional car , and azimuth A three-dimensional column vector, Indicates the linear speed of the omnidirectional car , and angular velocity The three-dimensional column vector of ;

[0032] The tracking error dynamics The expression formula is:

[0033]

[0034] in, Indicates the current state of the car, the superscript represents the transpose of a matrix, Represents a column vector The derivative of represents the reference trajectory of the car, Represents a column vector The derivative of represents the system matrix, , Indicates about and The diagonal matrix of , , , and Both represent system matrices, Indicates control input.

[0035] Preferably, the function formula for tracking the performance index in step S22 is:

[0036]

[0037] in, Represents tracking performance indicators, Tracking error The weighted quadratic form of , superscript represents the transpose of a matrix, and represents a 3D positive definite matrix, represents the control input, represents the integral variable;

[0038] The formula of the value function is:

[0039]

[0040] in, represents the value function;

[0041] The Hamiltonian equation is expressed as:

[0042]

[0043] in, represents the Hamiltonian function, Representation value function Tracking Error The partial derivative of and represents the system matrix.

[0044] As a preference, in step S23 HJB The equation is:

[0045]

[0046] in, represents the Hamiltonian function, represents the tracking error, represents the corresponding optimal control strategy, Represents the optimal value function for tracking error The partial derivative of .

[0047] Preferably, the step S24 includes the following formula:

[0048]

[0049] in, represents the optimal value function, represents the ideal evaluation network weight, with superscript represents the transpose of a matrix, represents the activation function vector, represents the approximation error, represents the number of neurons in the evaluation network;

[0050]

[0051] in, represents the optimal control strategy, Represents a 3D positive definite matrix The inverse, Represents the system matrix The transpose of Represents the activation function vector for tracking error The partial derivative of The approximation error The derivative of

[0052] according to Weierstrass Higher-order approximation theory, for optimal value functions and optimal control strategy Approximate and obtain the optimal value function The estimated value and optimal control strategy of An estimated value of

[0053] The optimal value function Estimated value of The formula is:

[0054]

[0055] The optimal control strategy Estimated value of The formula is:

[0056]

[0057] in, Represents the estimated value of the neural network weights.

[0058] Preferably, the step S25 includes the following formula:

[0059] The optimal value function The estimated value and optimal control strategy of Substituting the estimated value of into the Hamiltonian equation, we get the approximate Hamiltonian equation for:

[0060]

[0061] in, represents the tracking error, Represents the estimated value of the neural network weight, the superscript represents the transpose of a matrix, represents the system matrix, represents the estimated value of the current control strategy, Represents the system matrix The weighted quadratic form of , Represents the activation function vector Tracking Error The partial derivative of

[0062] definition HJB The residual term of the equation is the formula for the Bellman residual:

[0063]

[0064] in, represents the Bellman residual, represents the Bellman residual The quadratic form of

[0065] The formula for the neural network weight update law is:

[0066]

[0067] in, Represents the estimated value of the neural network weight The derivative of represents the parameter matrix, , represents the system matrix, represents a positive constant, represents the control input, Tracking error The weighted quadratic form of , and represents a 3D positive definite matrix.

[0068] Preferably, the formula of the neural network weight update law introducing the stability term in step S26 is:

[0069]

[0070] in, Represents the estimated value of the neural network weight after the introduction of the stabilization term The derivative of represents the stable term, , Represents the system matrix The weighted quadratic form of , and represents a positive constant, represents the parameter matrix, , Represents the activation function vector Tracking Error The partial derivative of represents the control input, Tracking error The weighted quadratic form of , and represents a 3D positive definite matrix, with the superscript represents the transpose of a matrix, Indicates a specified time interval, Indicated in The tracking error sampled at time , Indicated in The tracking error sampled at time .

[0071] Preferably, the analysis of the consistent final bounded stability of the omnidirectional car system in step S3 includes the following formula:

[0072] The continuous excitation condition is: the control input signal Bounded stability, if in the time interval The conditions for continuous incentives are:

[0073]

[0074] in, and represents a positive constant, represents the identity matrix of the corresponding dimension, represents the normalized parameter matrix, , represents the parameter matrix, , Represents the activation function vector Tracking Error The partial derivative of and represents the system matrix, Indicates control input, superscript represents the transpose of a matrix, represents the integration variable; such that:

[0075]

[0076] in, represents the Bellman residual, represents the maximum value of Bellman residual;

[0077] Select the entire omnidirectional car closed-loop system Lyapunov function for:

[0078]

[0079] in, represents a positive constant, represents the tracking error, represents the estimated error of the neural network weights, represents the transposed matrix of the estimated error of the neural network weights, , and taking its derivative we get:

[0080]

[0081] in, express Lyapunov function The derivative of Tracking error The derivative of Represents the neural network weight estimation error The derivative of HJB Equations and weight update laws, estimation errors of neural network weights The derivative is:

[0082]

[0083] in, represents a positive constant, represents the optimal control strategy, Represents the system matrix The weighted quadratic form of , represents the approximation error versus tracking error The derivative of represents the estimated error of the neural network weights, Represents the residual sum caused by the neural network approximation error. Substituting the above formula into Lyapunov Derivative of a function In the result:

[0084]

[0085] in, , , and is a positive constant, represents the mass of the car. When the following conditions are met, Lyapunov Derivative of a function :

[0086]

[0087] According to the tracking error and neural network weight estimation error The definition of Lyapunov The stability theorem shows that all closed-loop system signals of the omnidirectional car are consistently and ultimately bounded.

[0088] The beneficial effects of the above preferred solution are:

[0089] 1. Through the above technical solution, the kinematics and dynamics modeling of the physical omnidirectional car is carried out and converted into a more general strict feedback system to facilitate subsequent analysis.

[0090] 2. Through the sub-steps of step S2, the optimal control problem is transformed into a problem of solving partial differential equations, a neural network is introduced and the network weight update law is designed to achieve optimal control of the omnidirectional car trajectory tracking.

[0091] 3. When designing the weight update law of the neural network, the initial control strategy is restricted to admissible control by introducing a stabilization term to ensure the stability of the system and the optimality of the control strategy.

[0092] 4. Through continuous incentive conditions and Lyapunov Stability theorem, verify the optimal control algorithm of the trajectory tracking of the omnidirectional car, analyze the stability of the omnidirectional car system, and prove the effectiveness and reliability of this scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 Shown is a flow chart of an optimal control method for omnidirectional vehicle trajectory tracking based on reinforcement learning.

[0094] Figure 2 Shown is a simulation diagram of the trajectory tracking effect of the omnidirectional car.

[0095] Figure 3 Shown is the simulation diagram of the trajectory and linear velocity tracking error of the omnidirectional car.

[0096] Figure 4 Shown is a simulation diagram of the key control inputs of the omnidirectional car.

[0097] Figure 5 Shown is a simulation diagram of the performance function of the omnidirectional car. DETAILED DESCRIPTION

[0098] Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are only exemplary and are intended to explain the principles and spirit of the present invention, rather than to limit the scope of the present invention.

[0099] Embodiment 1, as Figure 1 As shown, an optimal control method for omnidirectional vehicle trajectory tracking based on reinforcement learning includes the following steps:

[0100] S1. Establish the trajectory kinematics model and dynamics model of the omnidirectional car;

[0101] S2. Based on the trajectory kinematics model and dynamics model of the omnidirectional car, design the optimal control algorithm for the trajectory tracking of the omnidirectional car based on neural network solution;

[0102] S3. Based on the trajectory tracking optimal control algorithm and continuous excitation conditions of the omnidirectional car, the uniformly ultimately bounded stability of the omnidirectional car system is analyzed, and the optimal control method for trajectory tracking of the omnidirectional car based on reinforcement learning is completed.

[0103] In this embodiment, the step S1 specifically includes the following formula:

[0104] An omnidirectional car with three DC motors as power devices is selected as the controlled object. The three DC motors respectively drive three Mecanum wheels evenly distributed at intervals of 120°. The omnidirectional motion of the car is achieved through the differential rotation of the wheels. The trajectory kinematic model and dynamic model of the omnidirectional car are modeled as follows:

[0105]

[0106] in, Indicates the position of the omnidirectional car , and azimuth The derivative of a three-dimensional column vector, Indicates the linear speed of the omnidirectional car , and angular velocity The three-dimensional column vector of Represents a three-dimensional column vector The derivative of Indicated by azimuth is the system matrix of parameters, , express dimensional constant matrix, , Indicates the mass of the car, represents the moment of inertia of the car, express dimensional state matrix, , represents the angular velocity, express dimensional constant matrix, , represents the average distance from the geometric center of the car to the wheel, represents the system control input three-dimensional column vector, Represents the motor torque constant;

[0107] The trajectory kinematics model and dynamics model of the omnidirectional car are organized to obtain a strict feedback form:

[0108]

[0109] in, Indicates the position of the omnidirectional car , and azimuth A three-dimensional column vector, Indicates the linear speed of the omnidirectional car , and angular velocity The three-dimensional column vector of represents the control input, , , , and Both represent system matrices, , , , , Indicates the wheel radius.

[0110] In this embodiment, step S2 specifically includes the following sub-steps:

[0111] S21. Based on the trajectory kinematic model and dynamic model of the omnidirectional car, a reference trajectory to be tracked by the omnidirectional car is set, and a tracking error is introduced and the tracking error dynamics is calculated;

[0112] S22. Dynamically define the omnidirectional car tracking performance index based on the tracking error, and obtain the Hamiltonian equation according to the value function definition;

[0113] S23. According to Hamilton's equation, we get HJB Equation;

[0114] S24. Introducing neural network HJB The optimal value function in the equation is approximated, and according to HJB The analytical relationship of the equation is used to update the control strategy;

[0115] S25. Definition HJB The residual term of the equation is used to design the neural network weight update law based on the residual normalized gradient descent method;

[0116] S26. Introduce a stabilization term into the neural network weight update law and obtain the neural network weight update law with the stabilization term to ensure that the control strategy always meets the conditions of admissible control and complete the design of the optimal control algorithm for omnidirectional car trajectory tracking based on reinforcement learning.

[0117] In this embodiment, the expression formula of the reference trajectory to be tracked by the omnidirectional vehicle in step S21 is:

[0118]

[0119] in, Indicates the reference position of the omnidirectional car , and the reference azimuth A three-dimensional column vector, Indicates the reference linear speed of the omnidirectional car , and the reference angular velocity A three-dimensional column vector, and Respectively and The derivative of , and All represent system matrices;

[0120] The tracking error Including tracking error and tracking error , and its calculation formula is:

[0121]

[0122] in, Indicates the position of the omnidirectional car , and azimuth A three-dimensional column vector, Indicates the linear speed of the omnidirectional car , and angular velocity The three-dimensional column vector of ;

[0123] The tracking error dynamics The expression formula is:

[0124]

[0125] in, Indicates the current state of the car, the superscript represents the transpose of a matrix, Represents a column vector The derivative of represents the reference trajectory of the car, Represents a column vector The derivative of represents the system matrix, , Indicates about and The diagonal matrix of , , , and Both represent system matrices, Indicates control input.

[0126] In this embodiment, the function formula for tracking the performance index in step S22 is:

[0127]

[0128] in, Represents tracking performance indicators, Tracking error The weighted quadratic form of , superscript represents the transpose of a matrix, and represents a 3D positive definite matrix, represents the control input, represents the integral variable;

[0129] The formula of the value function is:

[0130]

[0131] in, represents the value function;

[0132] The Hamiltonian equation is expressed as:

[0133]

[0134] in, represents the Hamiltonian function, Representation value function Tracking Error The partial derivative of and represents the system matrix.

[0135] In this embodiment, the step S23 HJB The equation is:

[0136]

[0137] in, represents the Hamiltonian function, represents the tracking error, represents the corresponding optimal control strategy;

[0138] According to the HJB equation and the stability condition, the optimal control strategy can be obtained as:

[0139]

[0140] in, represents the optimal control strategy; Represents a 3D positive definite matrix The inverse of Represents the system matrix The transpose of Represents the optimal value function Tracking Error The partial derivative of .

[0141] In this embodiment, step S24 includes the following formula:

[0142]

[0143] in, represents the optimal value function, represents the ideal evaluation network weight, , superscript represents the transpose of a matrix, represents the activation function vector, , represents the approximation error, represents the number of neurons in the evaluation network;

[0144]

[0145] in, represents the optimal control strategy, Represents a 3D positive definite matrix The inverse, Represents the system matrix The transpose of Represents the activation function vector for tracking error The partial derivative of The approximation error The derivative of

[0146] according to WeierstrassHigher-order approximation theory, for optimal value functions and optimal control strategy To approximate, when the number of neurons , then the approximation error , and get the optimal value function The estimated value and optimal control strategy of An estimated value of

[0147] The optimal value function Estimated value of The formula is:

[0148]

[0149] The optimal control strategy Estimated value of The formula is:

[0150]

[0151] in, Represents the estimated value of the neural network weights.

[0152] In this embodiment, step S25 includes the following formula:

[0153] The optimal value function The estimated value and optimal control strategy of Substituting the estimated value of into the Hamiltonian equation, we get the approximate Hamiltonian equation for:

[0154]

[0155] in, represents the tracking error, Represents the estimated value of the neural network weight, the superscript represents the transpose of a matrix, represents the system matrix, represents the estimated value of the current control strategy, Represents the system matrix The weighted quadratic form of , Represents the activation function vector Tracking Error The partial derivative of

[0156] definition HJB The residual term of the equation is the formula for the Bellman residual:

[0157]

[0158] in, represents the Bellman residual, represents the Bellman residual The quadratic form of

[0159] The formula for the neural network weight update law is:

[0160]

[0161] in, Represents the estimated value of the neural network weight The derivative of represents the parameter matrix, , represents the system matrix, represents a positive constant, represents the control input, Tracking error The weighted quadratic form of , and represents a 3D positive definite matrix.

[0162] In this embodiment, the formula of the neural network weight update law introducing the stability term in step S26 is:

[0163]

[0164] in, Represents the estimated value of the neural network weight after the introduction of the stabilization term The derivative of represents the stable term, , Represents the system matrix The weighted quadratic form of , and represents a positive constant, represents the parameter matrix, , Represents the activation function vector Tracking Error The partial derivative of represents the control input, Tracking error The weighted quadratic form of , and represents a 3D positive definite matrix, with the superscript represents the transpose of a matrix, Indicates a specified time interval, Indicated in The tracking error sampled at time , Indicated in The tracking error sampled at time .

[0165] The weight update law of the evaluation network shows that when the state of the controlled system shows a stable trend, the weight update law is responsible for optimizing the control strategy to approach the optimal control strategy; when the state of the controlled system shows a divergent trend, the stability term of the weight update law The system will be stabilized again, and on this basis, it will approach the optimal control strategy. Therefore, under this weight update law, any initial strategy can ensure the stability of the system, and the conditional restrictions of the allowable control can be lifted.

[0166] In this embodiment, the analysis of the consistent final bounded stability of the omnidirectional car system in step S3 includes the following formula:

[0167] The continuous excitation condition is: the control input signal Bounded stability, if in the time interval The conditions for continuous incentives are:

[0168]

[0169] in, and represents a positive constant, represents the identity matrix of the corresponding dimension, represents the normalized parameter matrix, , represents the parameter matrix, , Represents the activation function vector Tracking Error The partial derivative of and represents the system matrix, Indicates control input, superscript represents the transpose of a matrix, represents the integration variable; such that:

[0170]

[0171] in, represents the Bellman residual, represents the maximum value of Bellman residual;

[0172] Select the entire omnidirectional car closed-loop system Lyapunov function for:

[0173]

[0174] in, represents a positive constant, represents the tracking error, represents the estimated error of the neural network weights, represents the transposed matrix of the estimated error of the neural network weights, , and taking its derivative we get:

[0175]

[0176] in, express Lyapunov function The derivative of Tracking error The derivative of Represents the neural network weight estimation error The derivative of HJB Equations and weight update laws, estimation errors of neural network weights The derivative is:

[0177]

[0178] in, represents a positive constant, represents the optimal control strategy, Represents the system matrix The weighted quadratic form of , represents the approximation error versus tracking error The derivative of represents the estimated error of the neural network weights, Represents the residual sum caused by the neural network approximation error. Substituting the above formula into Lyapunov Derivative of a function In the result:

[0179]

[0180] in, , , and is a positive constant, represents the mass of the car. When the following conditions are met, Lyapunov Derivative of a function :

[0181]

[0182] According to the tracking error and neural network weight estimation error The definition of Lyapunov The stability theorem shows that all closed-loop system signals of the omnidirectional car are consistently and ultimately bounded.

[0183] Embodiment 2, this embodiment is by Matlab The method proposed in the present invention is verified by simulation.

[0184] The physical parameters of the omnidirectional car are set as follows: , , , , .

[0185] The reference trajectory is composed of a time-varying signal generate;

[0186] The initial position and initial speed of the omnidirectional car are randomly generated: , ;

[0187] Weight update parameters , , value function parameter , , the number of neurons in the neural network , activation function The choice of tracking error All quadratic combinations of .

[0188] The simulation results are as follows Figures 2 to 5 As shown, Figure 2 It shows the tracking effect of the omnidirectional car on the reference trajectory in a two-dimensional plane. Figure 3 The trajectory and linear velocity tracking errors of the omnidirectional car are shown. It can be seen that the tracking errors of both are bounded; Figure 4 The key control inputs of the omnidirectional car are shown. It can be seen that the control inputs are bounded, which shows that the proposed method is feasible in practical applications. Figure 5 It can be seen that with the iteration of training, the performance index function of the omnidirectional car decreases rapidly, which means that the integral sum of the tracking error and the control input is getting smaller. It is worth noting that because random noise is added during the training process to meet the continuous excitation condition, the effect presented is not a stable and smooth one. Therefore, the above simulation results show that the optimal control algorithm for trajectory tracking of the omnidirectional car based on reinforcement learning proposed by the present invention is effective.

[0189] The present invention solves the optimal control problem of omnidirectional car trajectory tracking and removes the conditional restriction that the initial strategy must be admissible control. It is also applicable to general strict feedback systems. Unlike the traditional gradient descent method, the present invention adds a stability term to the network weight update law according to the stability of the system, solving the problem of neural network training divergence caused by the initial strategy being non-admissible control.

[0190] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. An optimal control method for omnidirectional vehicle trajectory tracking based on reinforcement learning, characterized in that: The following steps are involved: S1. Establish the trajectory kinematics model and dynamics model of the omnidirectional car; S2. Based on the trajectory kinematics model and dynamics model of the omnidirectional car, design the optimal control algorithm for the trajectory tracking of the omnidirectional car based on neural network solution; S3. Based on the trajectory tracking optimal control algorithm and continuous excitation conditions of the omnidirectional car, the consistent final bounded stability of the omnidirectional car system is analyzed, and the optimal control of the trajectory tracking of the omnidirectional car based on reinforcement learning is completed; The step S2 specifically includes the following sub-steps: S21. Based on the trajectory kinematic model and dynamic model of the omnidirectional car, a reference trajectory to be tracked by the omnidirectional car is set, and a tracking error is introduced and the tracking error dynamics is calculated; S22. Dynamically define the omnidirectional car tracking performance index based on the tracking error, and obtain the Hamiltonian equation according to the value function definition; S23. According to Hamilton's equation, we get HJB Equation; S24. Introducing neural network HJB The optimal value function in the equation is approximated, and according to HJB The analytical relationship of the equation is used to update the control strategy; S25. Definitions HJB The residual term of the equation is used to design the neural network weight update law based on the residual normalized gradient descent method; S26. Introduce a stabilization term into the neural network weight update law, obtain the neural network weight update law with the stabilization term introduced, and complete the design of the optimal control algorithm for omnidirectional vehicle trajectory tracking based on reinforcement learning; The formula of the neural network weight update law introducing the stability term in step S26 is: in, Represents the estimated value of the neural network weight after the introduction of the stabilization term The derivative of represents the stable term, , Represents the system matrix The weighted quadratic form of , and represents a positive constant, represents the parameter matrix, , Represents the activation function vector Tracking Error The partial derivative of represents the control input, Tracking error The weighted quadratic form of , and represents a 3D positive definite matrix, with the superscript represents the transpose of a matrix, Indicates a specified time interval, Indicated in The tracking error sampled at time , Indicated in The tracking error sampled at time .

2. The optimal control method for omnidirectional vehicle trajectory tracking based on reinforcement learning according to claim 1 is characterized in that: In step S1, Includes the following formulas: An omnidirectional car with three DC motors as power devices is selected as the controlled object. The three DC motors respectively drive three Mecanum wheels evenly distributed at intervals of 120°. The omnidirectional motion of the car is achieved through the differential rotation of the wheels. The trajectory kinematic model and dynamic model of the omnidirectional car are modeled as follows: in, Indicates the position of the omnidirectional car , and azimuth The derivative of a three-dimensional column vector, Indicates the linear speed of the omnidirectional car , and angular velocity The three-dimensional column vector of Represents a three-dimensional column vector The derivative of Indicated by azimuth is the system matrix of parameters, , express dimensional constant matrix, , Indicates the mass of the car, represents the moment of inertia of the car, express dimensional state matrix, , represents the angular velocity, express dimensional constant matrix, , represents the average distance from the geometric center of the car to the wheel, represents the system control input three-dimensional column vector, Represents the motor torque constant; The trajectory kinematics model and dynamics model of the omnidirectional car are organized to obtain a strict feedback form: in, Indicates the position of the omnidirectional car , and azimuth A three-dimensional column vector, Indicates the linear speed of the omnidirectional car , and angular velocity The three-dimensional column vector of represents the control input, , , , and are system matrices, , , , , Indicates the wheel radius.

3. The optimal control method for omnidirectional vehicle trajectory tracking based on reinforcement learning according to claim 1 is characterized in that: The expression formula of the reference trajectory to be tracked by the omnidirectional car in step S21 is: in, Indicates the reference position of the omnidirectional car , and the reference azimuth A three-dimensional column vector, Indicates the reference linear speed of the omnidirectional car , and the reference angular velocity A three-dimensional column vector, and Respectively and The derivative of , and All represent system matrices; The tracking error Including tracking error and tracking error , and its calculation formula is: in, Indicates the position of the omnidirectional car , and azimuth A three-dimensional column vector, Indicates the linear speed of the omnidirectional car , and angular velocity The three-dimensional column vector of ; The tracking error dynamics The expression formula is: in, Indicates the current state of the car, the superscript represents the transpose of a matrix, Represents a column vector The derivative of represents the reference trajectory of the car, Represents a column vector The derivative of represents the system matrix, , Indicates about and The diagonal matrix of , , , and are system matrices, Indicates control input.

4. The optimal control method for omnidirectional vehicle trajectory tracking based on reinforcement learning according to claim 1 is characterized in that: The function formula for tracking the performance index in step S22 is: in, Represents tracking performance indicators, Tracking error The weighted quadratic form of , superscript represents the transpose of a matrix, and represents a 3D positive definite matrix, represents the control input, represents the integral variable; The formula of the value function is: in, represents the value function; The Hamiltonian equation is expressed as: in, represents the Hamiltonian function, Representation value function Tracking Error The partial derivative of and represents the system matrix.

5. The optimal control method for omnidirectional vehicle trajectory tracking based on reinforcement learning according to claim 1 is characterized in that: In step S23 HJB The equation is: in, represents the Hamiltonian function, represents the tracking error, represents the corresponding optimal control strategy, Represents the optimal value function for tracking error The partial derivative of .

6. The optimal control method for omnidirectional vehicle trajectory tracking based on reinforcement learning according to claim 1 is characterized in that: The step S24 includes the following formula: in, represents the optimal value function, represents the ideal evaluation network weight, with superscript represents the transpose of a matrix, represents the activation function vector, represents the approximation error, represents the number of neurons in the evaluation network; in, represents the optimal control strategy, Represents a 3D positive definite matrix The inverse, Represents the system matrix The transpose of Represents the activation function vector for tracking error The partial derivative of The approximation error The derivative of according to Weierstrass Higher-order approximation theory, for optimal value functions and optimal control strategy Approximate and obtain the optimal value function The estimated value and optimal control strategy of An estimated value of The optimal value function Estimated value of The formula is: The optimal control strategy Estimated value of The formula is: in, Represents the estimated value of the neural network weights.

7. The optimal control method for omnidirectional vehicle trajectory tracking based on reinforcement learning according to claim 6 is characterized in that: The step S25 includes the following formula: The optimal value function The estimated value and optimal control strategy of Substituting the estimated value of into the Hamiltonian equation, we get the approximate Hamiltonian equation for: in, represents the tracking error, Represents the estimated value of the neural network weight, the superscript represents the transpose of a matrix, represents the system matrix, represents the estimated value of the current control strategy, Represents the system matrix The weighted quadratic form of , Represents the activation function vector Tracking Error The partial derivative of definition HJB The residual term of the equation is the formula for the Bellman residual: in, represents the Bellman residual, represents the Bellman residual The quadratic form of The formula for the neural network weight update law is: in, Represents the estimated value of the neural network weight The derivative of represents the parameter matrix, , represents the system matrix, represents a positive constant, represents the control input, Tracking error The weighted quadratic form of , and represents a 3D positive definite matrix.

8. The optimal control method for omnidirectional vehicle trajectory tracking based on reinforcement learning according to claim 1 is characterized in that: The analysis of the consistent final bounded stability of the omnidirectional car system in step S3 includes the following formula: The continuous excitation condition is: the control input signal Bounded stability, if in the time interval The conditions for continuous incentives are: in, and represents a positive constant, represents the identity matrix of the corresponding dimension, represents the normalized parameter matrix, , represents the parameter matrix, , Represents the activation function vector Tracking Error The partial derivative of and represents the system matrix, Indicates control input, superscript represents the transpose of a matrix, represents the integration variable; such that: in, represents the Bellman residual, represents the maximum value of Bellman residual; Select the entire omnidirectional car closed-loop system Lyapunov function for: in, represents a positive constant, represents the tracking error, represents the estimated error of the neural network weights, represents the transposed matrix of the estimated error of the neural network weights, , and taking its derivative we get: in, express Lyapunov function The derivative of Tracking error The derivative of Represents the neural network weight estimation error The derivative of HJB Equations and weight update laws, estimation errors of neural network weights The derivative is: in, represents a positive constant, represents the optimal control strategy, Represents the system matrix The weighted quadratic form of , represents the approximation error versus tracking error The derivative of represents the estimated error of the neural network weights, Represents the residual sum caused by the neural network approximation error. Substituting the above formula into Lyapunov Derivative of a function In the result: in, , , and is a positive constant, represents the mass of the car. When the following conditions are met, Lyapunov Derivative of a function : According to the tracking error and neural network weight estimation error The definition of Lyapunov The stability theorem shows that all closed-loop system signals of the omnidirectional car are consistently and ultimately bounded.

Images