Optimal cooperative control method for preset performance of multiple intelligent vehicles under non-convex input constraint

Abstract

The invention relates to an optimal cooperative control method for preset performance of multiple intelligent vehicles under a non-convex input constraint, and belongs to the technical field of multi-agent cooperative control. The problems that non-convex input constraint processing is conservative, preset performance and optimal control are designed separately, and the timeliness of a traditional method is insufficient are solved. Designing a non-convex constraint operator to directly map, control and input to a feasible direction extreme value; constructing a distributed preset time observer to realize accurate estimation of the state of the leader; preset performance control is converted into an optimal control problem in combination with a time-varying performance function and error conversion; and solving an HJB equation based on a reinforcement learning iterative algorithm to approach an optimal control law. The conservative property of a traditional convex approximation method is broken through, and it is ensured that control input strictly meets the non-convex constraint; eliminating time uncertainty through preset time observation; fusing preset performance and optimal control to realize multi-target collaborative optimization; the reinforcement learning algorithm has global convergence and real-time performance, and is suitable for multi-vehicle cooperative control in a complex dynamic environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-agent cooperative control, and relates to an optimal cooperative control method for preset performance of multiple intelligent vehicles under non-convex input constraints. Background Art

[0002] Collaborative control technology for multi-agent systems serves as a core enabler for intelligent transportation, drone formations, and other fields. Its core goal is to achieve the emergence and coordination of swarm intelligence through distributed control strategies. Current research focuses primarily on idealized convex input constraint scenarios, such as actuator saturation constraints and symmetric speed limits. However, non-convex input constraints are common in practical engineering, such as the coupling constraints of joint torques in multi-joint manipulators and anisotropic speed limits on omnidirectional mobile robots. Such non-convex constraints result in complex geometric shapes in the feasible domain of control inputs, making traditional convex optimization-based control methods difficult to directly apply. Conservative convex approximation strategies are often employed, severely limiting the full potential of control performance.

[0003] In terms of transient performance control, preset performance control provides an effective framework for addressing issues such as system overshoot and convergence speed by designing a time-varying performance function to constrain tracking errors. However, existing methods suffer from two key flaws: First, the preset performance boundary typically uses a monotonically decreasing function, which makes it difficult to adapt to complex operating conditions such as periodic interference or sudden changes in the reference signal; second, the performance function is sensitive to initial conditions, and control failure may occur when the initial system error exceeds the preset range. Furthermore, traditional preset performance control lacks deep integration with optimal control theory, making it impossible to optimize indicators such as control energy consumption while ensuring transient performance.

[0004] Optimal control theory offers theoretical advantages in addressing multi-objective optimization and constraint satisfaction. However, its core challenge lies in the unavailability of analytical solutions due to the strong nonlinearity of the Hamilton-Jacobi-Bellman equation. Existing approximate solutions, such as adaptive dynamic programming and neural network approximation, generally face bottlenecks such as difficulty proving convergence and high computational complexity. For multi-agent systems, the curse of dimensionality further exacerbates the difficulty of designing optimal control strategies. Traditional methods, such as model predictive control, require high-frequency online optimization solutions, making them difficult to meet the real-time demands of large-scale vehicle collaboration.

[0005] More critically, existing technologies lack a theoretical understanding of the collaborative design of nonconvex input constraints and optimal control with preset performance. On the one hand, direct processing of nonconvex constraints often requires complex set operations, resulting in conservative control law design and high computational overhead. On the other hand, the coupling of preset performance functions with optimal control indicators lacks a systematic approach, making it difficult to achieve a balance between constraint satisfaction and performance optimization. These issues severely restrict the control performance of multi-intelligent vehicle systems in complex dynamic environments. There is an urgent need for new collaborative control methods that combine the ability to handle nonconvex constraints, guarantee preset performance, and achieve global optimality. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide an optimal collaborative control method for preset performance of multiple intelligent vehicles under non-convex input constraints, aiming to solve the collaborative tracking control problem of multiple intelligent vehicle systems in complex dynamic environments.

[0007] To address the challenges of nonconvex input constraints faced by intelligent vehicles in actual operation, such as the nonconvex feasible domain caused by anisotropic speed limits and the nonconvex constraint set caused by coupled actuator saturation, a nonconvex constraint operator is constructed to restrict the actuator's feedback input to the vector with the maximum magnitude in the same direction as the actuator, ensuring that all actual control inputs always fall within the corresponding nonconvex constraint set. Regarding performance control, the present invention constructs a tangent-type performance function and a distributed preset time observer to ensure that all following vehicles can accurately obtain the leader's status information under limited communication conditions.

[0008] The present invention combines nonlinear error transformation with the solution of the HJB equation, incorporates the transformation function and the system input into the performance index function at the same time, converts the preset time control problem into an optimal control problem, and approximates the optimal control law through the optimal controller iterative algorithm based on reinforcement learning, ensuring that the tracking error converges to the specified accuracy range within the preset time, and ensuring the collaborative optimality of the intelligent vehicle in the non-convex constraint set.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] An optimal cooperative control method for preset performance of multiple intelligent vehicles under non-convex input constraints includes the following steps:

[0011] S1: Construct a two-degree-of-freedom dynamic model for each follower vehicle and establish a multi-agent system consisting of N follower vehicles and one leader, where the dynamic equation of the i-th follower vehicle is:

[0012]

[0013] Among them, x i ∈R 4 is the state vector, A∈R 4×4is the system matrix, b∈R 4 is the control input matrix, is the control input subject to non-convex constraints;

[0014] S2: Design a non-convex constraint operator Φ(·) to restrict the control input to the non-convex constraint set Ω i Inside, the operator is defined as:

[0015]

[0016] in,

[0017] S3: Construct a distributed preset time observer so that the following vehicle can obtain the leader's status information within the preset time T:

[0018]

[0019] Where S∈R 4×4 For the leader system matrix, is the time-varying gain function, η,v∈R + is the adjustment parameter;

[0020] S4: Define tracking error And construct the preset performance function ζ(t):

[0021]

[0022] Among them, ε∈(0,1) is the preset accuracy;

[0023] S5: Convert the tracking error into an unbounded variable ξ via the tangent error transformation i :

[0024]

[0025] S6: Establish the coupled nonlinear Hamilton-Jacobi-Bellman equations:

[0026]

[0027] Among them, H,M∈R 4×4 is a symmetric positive definite matrix,

[0028] S7: Solve the HJB equation using an optimal controller iterative algorithm based on reinforcement learning, and approach the optimal control law by alternating between strategy evaluation and strategy improvement steps.

[0029] S8: Input the optimal control After being mapped by the non-convex constraint operator Φ(·), it acts on the following vehicle.

[0030] Furthermore, in S2, the non-convex constraint operator Φ i (x) Satisfy direction retention:

[0031]

[0032] Furthermore, in S3, the time-varying gain function μ(t) satisfies:

[0033]

[0034] where h,m∈Z + are preset parameters.

[0035] Furthermore, in S5, the tangent error transform satisfies the boundedness condition:

[0036] if and only if ||ξ i When ||<+∞, the tracking error is strictly limited to |e i (t)|<ζ(t)

[0037] Furthermore, in S7, the optimal controller iterative algorithm includes:

[0038] S71: Initialize permissible control inputs

[0039] S72: Update the value function by solving the Bellman equation

[0040]

[0041] S73: Update the control strategy according to the Hamiltonian condition:

[0042]

[0043] S74, repeat the iteration until where δ∈R + is the convergence threshold.

[0044] Furthermore, the preset performance function ζ(t) satisfies the boundary conditions:

[0045]

[0046] Furthermore, in S3, the communication topology satisfies a directed spanning tree with the leader as the root node, and the observation error satisfy:

[0047] ||ε i (t)||≤κμ -1 (t)e -βt ,κ,β∈R +.

[0048] Furthermore, in S5, the derivative of the tangent error transform is:

[0049]

[0050] Furthermore, the optimal control input Satisfying directional optimality:

[0051]

[0052] And through Φ i (·) After mapping, it satisfies

[0053] Furthermore, in S7, a deep neural network is used to approximate the value function:

[0054] J i (ξ i )=W T φ(ξ i )+ε(ξ i )

[0055] Where W∈R p is the weight vector, φ(·) is the activation function, and ε(·) is the approximation error.

[0056] The beneficial effects of the present invention are:

[0057] (1) The nonlinear constraint operator designed based on the geometric projection principle breaks through the conservative limitations of traditional convex approximation methods and directly maps the control input to the feasible direction extreme point of the original non-convex constraint set, significantly improving the utilization rate of the control input. It effectively solves the control feasibility problem of multiple intelligent vehicles in complex scenarios such as coupled actuator saturation and anisotropic speed limit, and avoids the control performance loss caused by convex relaxation.

[0058] (2) The distributed observer constructed based on the time-varying gain function can achieve accurate estimation of the leader's state information at the preset time under limited communication conditions. Compared with the traditional asymptotically convergent observer, it completely eliminates the time uncertainty and provides a reference signal with strict time domain guarantee for tracking control, thus ensuring the timeliness of collaborative control.

[0059] (3) Through the coupling design of time-varying performance function and nonlinear error transformation, the tracking error subject to preset boundary constraints is converted into an unbounded variable, and the Hamilton-Jacobi-Bellman equation is embedded to construct the optimal control problem, breaking through the limitation of the separate design of transient performance regulation and global energy consumption optimization in traditional methods, and realizing the multi-objective coordinated optimization of tracking accuracy, convergence speed and control energy consumption.

[0060] (4) The proposed iterative algorithm circumvents the problem of unavailable analytical solutions to the traditional HJB equation by alternately executing strategy evaluation and strategy improvement. It has been rigorously mathematically proven to have global convergence and can approximate the optimal control law that satisfies non-convex constraints online, significantly reducing computational complexity and meeting the real-time requirements of multi-vehicle collaboration.

[0061] (5) The designed non-monotonic performance function supports dynamic adjustment of time-varying boundaries, which can adapt to complex working conditions such as periodic interference or sudden changes in reference signals, and is compatible with symmetric and asymmetric error constraint scenarios, greatly improving the robustness and adaptability of the system in dynamic environments.

[0062] (6) The non-convex constraint operator strictly maintains the optimality of the control direction while limiting the input amplitude, avoiding the control direction deviation problem caused by the traditional truncation method, and ensuring the trajectory tracking accuracy and stability of multi-vehicle collaborative motion control.

[0063] (7) Through the collaborative architecture of distributed preset time observer and local optimal control law, the high-dimensional multi-vehicle coordination problem is decoupled into low-dimensional sub-problems and solved in parallel, which significantly reduces the computational complexity and provides feasibility support for the engineering application of large-scale vehicle clusters.

[0064] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0066] Figure 1 This is the vehicle force analysis diagram;

[0067] Figure 2 It is a two-degree-of-freedom bicycle model;

[0068] Figure 3 This is a vehicle tire side slip analysis diagram;

[0069] Figure 4 is a non-convex constraint operator;

[0070] Figure 5 This is the technical roadmap of the present invention;

[0071] Figure 6 This is a flowchart of the iterative algorithm for the optimal controller based on reinforcement learning. DETAILED DESCRIPTION

[0072] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0073] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0074] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0075] The present invention constructs a non-convex constraint operator for the non-convex constraints of the input. By dynamically projecting the feedback input of the actuator to the maximum amplitude boundary of the feasible direction, it ensures that all actual controls strictly meet the original non-convex constraint conditions and the consistency of the original control direction.

[0076] A distributed fixed-time observer is constructed, whose convergence time can be customized by the user, so that all intelligent vehicles can observe the leader's status information within a fixed time, saving communication resources.

[0077] A monotonically decreasing bounded auxiliary function related to the preset time and the preset accuracy is constructed. By incorporating this transformation function and the system input into the performance index function, the preset time control problem is transformed into an optimal control problem.

[0078] A reinforcement learning-based optimal controller iteration algorithm is proposed. After initialization, the algorithm relies on iteratively computing the coupled nonlinear game Bellman equation and iterative controller updates until the convergence error approaches 0. The relevant convergence proof of the algorithm is also given.

[0079] The present invention is designed to be composed of the following steps:

[0080] In order to deal with the problem of non-convex input constraints, a non-convex constraint operator is designed to transform the feedback input into a vector with the maximum magnitude in the same direction as it.

[0081] A distributed preset time observer is constructed so that the follower vehicle can accurately obtain the state information of the leader vehicle within the preset time and use it to set the tracking error.

[0082] A preset performance function is designed for a preset time and a preset accuracy, and an auxiliary error conversion function is designed for the preset performance function and tracking error. By constructing the HJB equation for the auxiliary error conversion function and the control input, the preset performance control problem is transformed into an optimal control problem. The solution of the HJB equation is learned through the optimal controller iterative algorithm based on reinforcement learning, thereby obtaining the optimal control law.

[0083] To achieve the above object, the technical solution of the present invention is as follows:

[0084] The optimal collaborative control scheme for multiple intelligent vehicles with preset performance under non-convex input constraints designed by the present invention mainly consists of the following parts: the non-convex constraint operator is input to the maximum amplitude boundary of the feasible direction through dynamic projection feedback, which strictly limits the control input of the actuator to the allowable control set that satisfies the non-convex constraint while maintaining the optimality of the control direction; a distributed preset time observer is constructed to ensure that all following vehicles accurately obtain the leader's state information within the preset time, providing an accurate reference for collaborative tracking; the preset performance function and the auxiliary error conversion function convert the tracking error into a bounded error variable through nonlinear mapping, and construct the HJB equation. The optimal control law is obtained by approximating the nonlinear solution of the HJB equation through the optimal controller iterative algorithm based on reinforcement learning, and the convergence of the algorithm is guaranteed by rigorous mathematical proof; the above control scheme is combined with the reinforcement learning framework to form an optimal collaborative control method based on non-convex constraints and preset performance, which can realize high-precision tracking and optimal control performance of the multi-intelligent vehicle system in complex environments.

[0085] Part I Model establishment and system description

[0086] A two-degree-of-freedom dynamic model

[0087] To accurately characterize the vehicle's handling stability and suspension smoothness during driving, a kinematic model that can simultaneously reflect both longitudinal and lateral dynamic characteristics must be established. This paper uses a two-degree-of-freedom vehicle dynamics model as the basic modeling framework and makes the following assumptions:

[0088] Assumption 1: Ignore the dynamic coupling effect of the steering system and suspension system; only consider the linear characteristics of the tire cornering stiffness (the slip angle is in the linear operating range); assume that the vertical load on the tire is uniformly distributed, and ignore the impact of load transfer on the tire force; do not consider aerodynamic effects (including longitudinal aerodynamic drag and lateral aerodynamic side force); based on geometric simplification conditions, assume that the wheelbase is much smaller than the turning radius, so that the single-vehicle model can be used to describe the overall vehicle motion characteristics.

[0089] According to the assumption, combined Figure 1 and Figure 2 The vehicle is mechanically analyzed along the xyz axes, and according to Newton's second law, it is obtained:

[0090]

[0091] in is the inertial acceleration in the y-axis direction at the center of gravity of the vehicle; F y1 and F y2 They represent the components of the force acting on the front and rear wheels of the vehicle in the y-axis direction; F x1 and F x2 They represent the x-axis components of the forces acting on the front and rear wheels of the vehicle, respectively. L1 and L2 represent the distances from the front and rear wheels to the center of mass, respectively.

[0092] The inertial acceleration is generated by the combined effect of the vehicle's inertial acceleration along the y-axis and the centripetal acceleration generated by the vehicle's rotation around the center of rotation. Therefore, the inertial acceleration of the vehicle can be deduced:

[0093]

[0094] Substituting formula (2) into formula (1), the dynamic equations in the x-axis and y-axis directions are as follows:

[0095]

[0096] according to Figure 3 The vehicle tire slip model shown in the figure can obtain the front wheel slip angle as:

[0097]

[0098] Where θ1 represents the front wheel steering angle, Represents the angle between the front wheel velocity vector and the vehicle's longitudinal axis.

[0099] For the same reason, since the rear wheel steering angle is 0, the rear wheel slip angle can be obtained as:

[0100]

[0101] in Represents the angle between the rear wheel velocity vector and the vehicle's longitudinal axis.

[0102] The lateral force on the front wheels of the vehicle can be expressed as:

[0103]

[0104] The rear wheel lateral force can be expressed as:

[0105]

[0106] Among them, the proportional constants K1 and K2 are the cornering stiffness of the front and rear wheels of the vehicle, respectively.

[0107] After substituting and sorting, let u = θ1 be the controller input, and we can get the following state space equation of the two-degree-of-freedom vehicle dynamics model:

[0108]

[0109] The specific expressions of matrices A and b are as follows:

[0110]

[0111]

[0112] in

[0113] B. System Description

[0114] Consider a multi-agent system consisting of N follower intelligent vehicles and 1 leader (reference signal), where the dynamics equation of the i-th agent is:

[0115]

[0116] Since the driving force of the intelligent agent in different directions may have different magnitudes, the control input u of the follower intelligent vehicle is i is restricted to a non-convex set, i.e. Ω i is a non-empty bounded closed set and 0∈Ω i .

[0117] The dynamic equation of the leader is given by the following integral model:

[0118]

[0119] where f = [0 1 0 1] T ,x0=[q1 q2 ... q N ] T The S matrix expression is as follows:

[0120]

[0121] The local communication relationship between multi-agent systems satisfies the graph theory topology, and the Laplace matrix is ​​defined as L = DA. The matrix is ​​defined as The matrix B reflects the follower information that directly communicates with the leader. To facilitate subsequent design, some important assumptions and lemmas are listed below:

[0122] Assumption 2 Assume that the matrix pair (A, b) is controllable and the leader system is observable.

[0123] Assumption 3: The communication topology graph contains a directed spanning tree with the leader as the root node.

[0124] Assumption 4: The leader input signal u0 is unknown but has an upper bound. That is, for all t∈[t0,∞), there exists an unknown constant ω such that |u0|<ω.

[0125] Lemma 1: Consider a system defined as follows:

[0126]

[0127] x(0) = x0, where x(t)∈R n , assuming that the origin is an equilibrium point of the system, if there exists a continuously differentiable, positive definite, fundamentally unbounded function V(x) that satisfies:

[0128]

[0129] Therefore, the equilibrium point of system (15) satisfies finite-time stability, and the convergence time T satisfies (17):

[0130]

[0131] Lemma 2: The time-varying function μ(t) is as follows:

[0132]

[0133] Consider system (15), if there exists a positive real number b such that V(x) satisfies:

[0134] V(0)=0,V(x)>0 (19)

[0135]

[0136] Then the system (15) is stable within the preset time T, and V(x)≡0 within t∈[t0+T,∞].

[0137] Assumption 5: Expected tracking trajectory q i (t) and its first-order derivative are bounded.

[0138] The tracking error is defined as follows:

[0139]

[0140] Among them, q i (t) represents the desired target signal, Represents the observation signal of the observer.

[0141] The control objective of the present invention is to develop an optimal controller iterative algorithm based on reinforcement learning through error conversion under the premise of satisfying non-convex constraints, and to achieve the following control objectives:

[0142] All control inputs of O1 are strictly limited to the admissible control set to ensure that the inputs satisfy the non-convex constraints.

[0143] The distributed preset time observer constructed by O2 enables multiple intelligent vehicles to obtain the desired tracking signals within local communication and preset time;

[0144] In the O3 closed-loop system, all error signals e(t) converge to a tight set of artificially set precision within a preset time, and all signals remain bounded in the entire time domain.

[0145] Part II Description of Important Technical Means

[0146] A non-convex constraint operator

[0147] In order to handle the problem that the system input is subject to non-convex constraints, the present invention defines the following non-convex constraint operator.

[0148] Definition 1 Non-convex constraint operator Φ i (·)satisfy:

[0149]

[0150] in

[0151] Non-convex constraint operator Φ i (x) Feedback input u i Transformed into an extreme vector Φ in the same direction i (u i ), so that for all 0≤ρ≤1, there is ρΦ i (u i )∈Ωi , that is, the actual control input will always be within the non-convex constraint set Ω without any convex assumption. i The control input direction is adjustable and the variable of the control quantity in any direction is guaranteed to be not zero. Figure 4 A specific example is given to illustrate Φ more intuitively. i The operation mechanism of (x) and its effect under non-convex constraints.

[0152] B error conversion auxiliary function

[0153] The present invention first introduces the following preset performance function:

[0154]

[0155] Where ∈∈(0, 1) is the preset accuracy, and T is the preset time.

[0156] And based on the preset performance function ζ(t) and the tracking error e i (t) Construct the following tangent error transform:

[0157]

[0158] where ξ=[ξ1,ξ2,...,ξ N ] T ∈R N×4 , and there is e i (0)<ζ(0).

[0159] The derivative of (24) is:

[0160]

[0161] The expression of γ is as follows:

[0162]

[0163] From formula (25), we can see that if the function ξ i Bounded and initial error e i (0)Satisfy-ζ(0) <e i (0)<ζ(0), then according to the properties of the tangent function, we can get -ζ <e i <ζ. According to the properties of the preset performance function ζ(t) in formula (23), we can further obtain Have-∈ <e i <∈. This shows that as long as -ζ(0) is satisfied <e i (0)<ζ(0), as long as ξ i The boundedness of e i (t) will converge to the compact set Ω within the preset time Te middle.

[0164] C Optimal controller iterative algorithm based on reinforcement learning

[0165] The optimal controller iteration algorithm used in this paper is a value iteration method based on reinforcement learning theory. It is used to solve the nonlinear optimal control solution approximating the HJB equation. The algorithm achieves progressive optimization of the control strategy by alternating between two key steps: strategy evaluation and strategy improvement.

[0166] In the strategy evaluation stage, the value function of the current control strategy is iteratively estimated by constructing the Bellman equations; in the strategy improvement stage, the control input is gradient updated using the necessary conditions of the first-order derivative of the HJB equation.

[0167] The algorithm employs an offline learning approach, constructing an experience replay buffer using historical system data, utilizing batch sampling techniques to improve data utilization efficiency, and integrating target network technology to ensure the stability of the learning process. Theoretical analysis demonstrates that this policy iteration algorithm has strict mathematical convergence guarantees, asymptotically approximating the true solution of the HJB equation and ultimately obtaining an optimal control strategy that meets the preset performance requirements. Compared to traditional online learning methods, offline policy iteration algorithms offer higher data efficiency and better convergence stability, making them suitable for control scenarios with stringent safety and real-time requirements, such as multi-intelligent vehicle systems.

[0168] Part III Design of Distributed Preset Time Observer

[0169] APreset time observer

[0170] According to Lemma 2, we construct the following distributed preset time observer:

[0171]

[0172] Where η = η1 + η2, and the observer error is defined as:

[0173]

[0174] B. Convergence Proof

[0175] If assumptions 3-4 hold, and the leader dynamic model described by Equation (13) is considered, the distributed observer (27) can ensure that the observation error (28) will tend to zero at an exponential rate, thereby providing each intelligent vehicle with an accurate estimate of the leader's state.

[0176] prove:

[0177] After deriving (28), the distributed observer can be reformulated as follows:

[0178]

[0179] Choose the Lyapunov function as:

[0180]

[0181] Where P is a diagonal positive definite matrix, and the derivative of V1(t) is:

[0182]

[0183] make And choose the parameter k1 so that It can be organized as follows:

[0184]

[0185] According to the finite time theorem, we can further obtain:

[0186]

[0187]

[0188] From this, it is inferred that ε i It is uniformly bounded on the interval [0, t0+T].

[0189] Step 2: For Observer error ε i Will remain at 0

[0190] Proof: According to inequality (20), we can get:

[0191]

[0192] To ensure that x0(t) is consistent with The continuity at the specified time t = t0 + T, and the subsequent verification of the continuity at the time t = t0 + T, lead to the conclusion that: ε i At this specific moment, both V(t) and V1(t) exhibit continuity. This observation not only confirms that the function achieves seamless continuity at the moment t = t0 + T, but also marks a key conclusion with important implications for the dynamics of the system.

[0193]

[0194] Therefore, from the squeeze theorem we can infer that:

[0195]

[0196] This means that for every moment in [t0+T,∞], V1(t) is constant to zero. Therefore, the observation error ε i(t always remains zero. Therefore, it can be concluded that the state observer proposed in the present invention can accurately estimate the state information of the leader within the preset time T when appropriate parameters are selected.

[0197] The proof is complete.

[0198] Part IV Design of Optimal Controller with Fixed-Time Preset Performance

[0199] A performance function definition

[0200] According to the error conversion auxiliary function ξ i As long as ξ i The boundedness of the tracking error e i (t) will converge to the preset accuracy range within the preset time, so the performance index function of the intelligent vehicle is defined as:

[0201]

[0202] Among them L i =ξ i T Hξ i +u i (τ) T Mu i (τ) is the utility function, and H and M are symmetric positive definite matrices. The appropriate selection of H and M depends on the specific goals and requirements of the control application. For example, by selecting H with a larger eigenvalue compared to M, the system state trajectory can converge to the required accuracy more quickly.

[0203] The present invention can convert the preset performance control problem into an optimal control problem by introducing the error conversion function (24) into the performance index function (38), and find the optimal control strategy within the preset time through the reinforcement learning algorithm, thereby achieving the preset time convergence to the preset accuracy. Compared with the existing technology, the present invention uses the adaptive optimization capability of reinforcement learning to i (t)=Φ i (α i )(α i is the control input before processing), while strictly satisfying the non-convex input constraints, it significantly reduces the conservatism of traditional methods and ensures that the system tracking error converges to the specified accuracy range within the preset time, breaking through the limitation of traditional preset performance control that can only guarantee asymptotic convergence.

[0204] B coupled nonlinear HJB equation

[0205] Define the optimal value function:

[0206]

[0207] Purpose: Design control input ui Let the value function J * (ξ i ) is minimized and the constraint operator Φ i (·) Make the input satisfy the admissible control.

[0208] According to equation (25), taking the time derivative of (39), the following coupled nonlinear HJB equation can be derived:

[0209]

[0210] in

[0211] The coupled nonlinear game Bellman equation is:

[0212]

[0213] According to the first-order necessary conditions of the Hamiltonian equation, the optimal control input that satisfies the non-convex constraints can be derived as:

[0214]

[0215] Defined by the non-convex constraint operator, it is obvious that the optimal control input is constrained within the prescribed limit, that is, u i ∈Ω i Because the error system and the agent value function are affected by the input of neighboring agents, by considering the optimal performance, the present invention can achieve tracking control under input-constrained conditions and reveals the impact of interactions between agents on the performance function.

[0216] Part V Optimal Controller Iteration Algorithm Based on Reinforcement Learning

[0217] The present invention proposes an optimal controller iterative algorithm based on models and reinforcement learning theory. The algorithm relies on the iterative calculation process of the value function in the HJB equation and the calculation process of the first-order necessary conditions of the HJB equation. Specifically, the coupled game Bellman equations (40) and equation (41) are used until the current control input no longer changes. The convergence proof of the algorithm will be given later.

[0218] Algorithm 1: Optimal controller iterative algorithm based on reinforcement learning

[0219] ① Initialize the permissible control input

[0220] ②Use the following coupled nonlinear game Bellman equations to solve the iterative value function

[0221]

[0222] ③ The optimal controller is iteratively updated, and the control input is updated using the following formula

[0223]

[0224] ④Return to step ② until The control error is small enough.

[0225] Lemma 3: In step ②, based on the current control input Solving the value function And meet:

[0226]

[0227] This means It is the control input The value function below.

[0228] In step ③, the control input is updated by minimizing the Hamiltonian function, namely:

[0229]

[0230] because is Therefore, according to the Bellman optimality principle, the value function shows a decreasing trend, that is:

[0231]

[0232] The optimal controller iterative algorithm proposed in this invention is transformed by the constraint operator mapping to obtain the initial control input. is admissible, and all intelligent vehicles simultaneously execute the optimal controller iterative algorithm based on reinforcement learning, then:

[0233] For all k>0, Still an admissible iterative value function is monotonically non-increasing and converges to the optimal value function Right now

[0234]

[0235] Iteration control input Converge to the optimal control input for the response Right now

[0236]

[0237] prove:

[0238] According to the definition of constraint operator, the first point is easy to prove Still an allowable control input.

[0239] According to the performance function definition, the value function is a positive definite function, so the Lyapunov function is selected as:

[0240]

[0241] From Lemma 3, we know that:

[0242]

[0243] Because J i (ξ i (t)) is monotonically non-increasing and positively bounded, so by the monotone boundedness criterion, the sequence Single challenge does not increase and has a lower bound, so it converges to a certain limit Therefore, the second point is proved.

[0244] The iterative algorithm is input by the optimal control:

[0245]

[0246] as well as and Easy to obtain iterative control input Converge to the optimal control input

[0247] Figure 5 This is the technical roadmap of the present invention;

[0248] Figure 6 This is a flowchart of the iterative algorithm for the optimal controller based on reinforcement learning.

[0249] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.

Claims

1. An optimal collaborative control method for multiple intelligent vehicles with preset performance under non-convex input constraints, characterized by: The following steps are involved: S1: Construct a two-degree-of-freedom dynamic model for each follower vehicle and establish a multi-agent system consisting of N follower vehicles and one leader, where the dynamic equation of the i-th follower vehicle is: Among them, x i ∈R 4 is the state vector, A∈R 4×4 is the system matrix, b∈R 4 is the control input matrix, is the control input subject to non-convex constraints; S2: Design a non-convex constraint operator Φ(·) to restrict the control input to the non-convex constraint set Ω i Inside, the operator is defined as: in, S3: Construct a distributed preset time observer so that the following vehicle can obtain the leader's status information within the preset time T: Where S∈R 4×4 For the leader system matrix, is the time-varying gain function, η,v∈R + To adjust the parameters; S4: Define tracking error And construct the preset performance function ζ(t): Among them, ε∈(0,1) is the preset accuracy; S5: Convert the tracking error into an unbounded variable ξ via the tangent error transformation i : S6: Establish the coupled nonlinear Hamilton-Jacobi-Bellman equations: Among them, H,M∈R 4×4 is a symmetric positive definite matrix, S7: Solve the HJB equation using an optimal controller iterative algorithm based on reinforcement learning, and approach the optimal control law by alternating between strategy evaluation and strategy improvement steps. S8: Input the optimal control After being mapped by the non-convex constraint operator Φ(·), it acts on the following vehicle.

2. The optimal collaborative control method for pre-set performance of multiple intelligent vehicles under non-convex input constraints according to claim 1 is characterized by: In S2, the non-convex constraint operator Φ i (x) Satisfy direction retention:

3. The optimal coordinated control method for preset performance of multiple intelligent vehicles under non-convex input constraints according to claim 1 is characterized by: In S3, the time-varying gain function μ(t) satisfies: where h,m∈Z + are preset parameters.

4. The optimal collaborative control method for preset performance of multiple intelligent vehicles under non-convex input constraints according to claim 1 is characterized by: In S5, the tangent error transform satisfies the boundedness condition: if and only if ||ξ i When ||<+∞, the tracking error is strictly limited to |e i (t)|<ζ(t)∣e.

5. The optimal collaborative control method for preset performance of multiple intelligent vehicles under non-convex input constraints according to claim 1 is characterized by: In S7, the optimal controller iterative algorithm includes: S71: Initialize permissible control inputs S72: Update the value function by solving the Bellman equation S73: Update the control strategy according to the Hamiltonian condition: S74, repeat the iteration until where δ∈R + is the convergence threshold.

6. The optimal collaborative control method for preset performance of multiple intelligent vehicles under non-convex input constraints according to claim 1 is characterized by: The preset performance function ζ(t) satisfies the boundary conditions:

7. The optimal collaborative control method for preset performance of multiple intelligent vehicles under non-convex input constraints according to claim 1 is characterized by: In S3, the communication topology satisfies a directed spanning tree with the leader as the root node, and the observation error satisfy: ||e i (t)||≤km -1 (t)e -βt ,κ,β∈R + 。 8. The optimal coordinated control method for preset performance of multiple intelligent vehicles under non-convex input constraints according to claim 1 is characterized by: In S5, the derivative of the tangent error transform is:

9. The optimal collaborative control method for preset performance of multiple intelligent vehicles under non-convex input constraints according to claim 1, characterized in that: The optimal control input Satisfying directional optimality: And through Φ i (·) After mapping, it satisfies 10. The optimal cooperative control method for preset performance of multiple intelligent vehicles under non-convex input constraints according to claim 1, characterized in that: In S7, a deep neural network is used to approximate the value function: J i (x) i )=W T f(x i )+e(x i ) Where W∈R p is the weight vector, φ(·) is the activation function, and ε(·) is the approximation error.

Images