Adaptive integral sliding mode cooperative queue control strategy based on reinforcement learning
By adopting an adaptive integral sliding mode cooperative queue control strategy based on reinforcement learning, the problem of neglecting energy efficiency in traditional methods is solved, and the stability of the queue and the minimization of energy consumption are achieved, thereby improving the overall performance and security of the queue.
Patent Information
- Application Number
- CN202610050517.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-14
- Publication Date
- 2026-03-17
AI Technical Summary
Traditional cooperative queue control methods typically only consider queue stability and tracking error accuracy in their design, while ignoring energy efficiency, making it difficult to minimize energy consumption in complex dynamic environments.
An adaptive integral sliding mode cooperative queue control strategy based on reinforcement learning is adopted. The neural network weights are adjusted online through a commentator-actor network architecture to optimize control performance. Combined with the queue integral sliding surface, stability and optimal control are achieved.
While ensuring the internal stability and control performance of the queue, it improves energy efficiency and enhances the safety and efficiency of the queue.
Smart Images

Figure CN121680086A_ABST
Abstract
Description
[0001] The application discloses a reinforcement learning-based adaptive integral sliding mode cooperative queue control strategy, belongs to the intelligent control field, and mainly realizes the stability and optimal control performance of the whole queue by adjusting neural network weights online under the conditions of actuator saturation, parameter uncertainty and external disturbance. TECHNICAL FIELD
[0002] The application belongs to the intelligent control field and mainly relates to a reinforcement learning-based adaptive integral sliding mode cooperative queue control strategy. BACKGROUND
[0003] In current complex industrial systems, with the continuous improvement of system scale and automation degree, the problems of operation efficiency, energy consumption control and system reliability are increasingly prominent, and higher requirements for intelligent management and cooperative control of the system are put forward. As a core control method of multi-agent system, cooperative queue control can effectively improve the overall performance of the system, guarantee the operation consistency and reduce the overall energy consumption, and has important significance for realizing an efficient and reliable intelligent system. However, the existing cooperative queue control methods still face many challenges in control performance and robustness when facing complex dynamic environments and internal disturbances. First, there are parameter uncertainties in the system and various random disturbances exist outside, which will significantly affect the synchronization accuracy and stability of the queue. Secondly, actuator saturation exists universally in actual systems, and its physical output limit may cause the system performance to decline or even become unstable. In addition, the traditional cooperative control method usually only focuses on the stability and tracking accuracy of the system in the design, and ignores the energy efficiency in the control process, so it is difficult to guarantee the performance while realizing energy optimization. In recent years, reinforcement learning as an intelligent control method with self-learning ability provides a new idea for the cooperative optimization problem of complex systems. Among them, the critic-actor network architecture can learn and approximate the optimal control law online by executing the control strategy through the actor network and evaluating the system performance through the critic network, which lays a technical foundation for realizing the adaptive cooperative control of multi-agent systems in uncertain environments. In summary, in order to meet the higher standards and requirements of modern industrial systems for cooperative control, the existing cooperative queue control methods need to be further improved and perfected to realize high-performance, high-energy efficiency and high-robustness control of the system in complex dynamic environments. SUMMARY
[0004] The purpose of the present application is to solve the problem that the traditional cooperative queue control method generally only considers the stability of the queue and the accuracy of the tracking error in design, ignores the key performance indicator of energy efficiency, and often cannot ensure the minimization of energy consumption, thereby limiting its application range and effect. The present application provides a self-adaptive integral sliding mode cooperative queue control strategy based on reinforcement learning.
[0005] A self-adaptive integral sliding mode cooperative queue control strategy based on reinforcement learning, characterized by a critic-actor network architecture based on reinforcement learning, which executes the control strategy through the actor network and evaluates the pros and cons of the strategy through the critic network, and adjusts the neural network weights online through reinforcement learning to achieve the stability of the entire queue and the optimal control performance. The control strategy comprises the following steps:
[0006] Step one, establish a queue system model with uncertainty, external disturbance and actuator saturation:
[0007] Queue system model
[0008]
[0009] Where i is the serial number, x i (t) and v i (t) represent the position and speed of the ith agent, respectively, t represents time, represents a column vector containing position, speed and time, is the uncertainty and external disturbance suffered by the system, u i (t) is the control input, 0 i ≤1 represents the degree of actuator saturation, and ε ui (t) represents the approximation error of the saturation function.
[0010] Step two, establish the cooperative queue tracking error:
[0011] Cooperative queue tracking error
[0012]
[0013] Where, represents the tracking error of the position, and are intermediate variables, d i (t)=x i-1 (t)-x i (t) is the adjacent spacing of the agent, is the safety distance when stationary, e represents a natural constant, represents the headway, v0(t) represents the speed of the leader, represents the tracking error of the speed, β represents the limiting value of t as it approaches 0. i It is a constant greater than zero.
[0014] Step 3: Establish a network of reinforcement learning commentators:
[0015] Reinforcement Learning Commentator Network
[0016]
[0017] in, Represents the cost function. Let τ represent the integral from time 0 to positive infinity, where τ is the variable of integration. Represents the discount factor. u(t)=[u1(t),u2(t),...,u i (t)] T , and It is a positive definite matrix.
[0018]
[0019] in, ξ represents the estimated value of the cost function. c (e x ) is the activation function of the commentator network, W c These are the ideal weights for the commentator's neural network. It is W c The estimated value, ε c It is the approximation error of the commentator's neural network.
[0020] Commentator Neural Network Update Law
[0021]
[0022] in, The learning rate represents the commentator network.
[0023] Step 4: Establish a reinforcement learning actor network:
[0024] Reinforcement learning actor network
[0025]
[0026] in, W is the activation function of the actor network. ai These are the ideal weights for the actor's neural network. and They are With Wai The estimated value, ε ai It is the approximation error of the actor's neural network.
[0027] Actor Neural Network Update Law
[0028]
[0029] in, Represents the dynamic learning rate. κ ai ε1 and ε2 are both constants greater than zero. It is the expected cumulative cost. It is a constant greater than zero.
[0030] Step 5: Establish the queue integration sliding surface:
[0031] Integral sliding surface
[0032]
[0033] Among them, s i (t) is the sliding surface. Let δ, λ1, and λ2 represent the integral from time 0 to time t, where δ, λ1, and λ2 are all constants greater than zero.
[0034]
[0035] Among them, S i (t) is the coupled sliding surface, 0 < q ≤ 1, and N is the number of agents.
[0036] Step 6: Establish an adaptive integral sliding mode cooperative queue control strategy based on reinforcement learning:
[0037] Adaptive control parameter update law
[0038]
[0039] Where, γ ai γ ui , σ ai and σ ui It is a constant greater than zero. and They are ε ai With ε ui The estimated value.
[0040] Cooperative queue control law
[0041]
[0042] Where ρ, η, ζ, α, and k are all positive constants, || is the absolute value, and tanh() is the hyperbolic tangent function.
[0043] Effects of the embodiments of the present invention:
[0044] The beneficial effect of this invention lies in its effective solution to the problem that traditional cooperative queue control methods typically only consider queue stability and tracking error accuracy in their design, while neglecting energy efficiency, a key performance indicator, and often failing to ensure minimal energy consumption. This invention ensures the internal stability and overall queue stability of the entire queue, while achieving optimal control performance, improving queue efficiency, and enhancing queue security. Attached Figure Description
[0045] Figure 1 This is a flowchart illustrating the cooperative queue control method described in Specific Implementation Method 1. Detailed Implementation
[0046] Specific implementation method one: Combining Figure 1 This embodiment describes an adaptive integral sliding mode cooperative queue control strategy based on reinforcement learning. The cooperative queue control strategy includes the following steps:
[0047] Step 1: Establish a queue system model with uncertainties, external disturbances, and actuator saturation:
[0048] Queue system model
[0049]
[0050] Where i is the sequence number, x i (t) and v i (t) represents the position and velocity of the i-th agent, respectively, and t represents time. This represents a column vector containing position, velocity, and time. It refers to the uncertainties and external disturbances that the system experiences. i (t) is the control input, 0 < θ i ≤1 represents the degree of actuator saturation, ε ui (t) represents the approximation error of the saturation function.
[0051] Step 2: Establish a collaborative queue to track errors:
[0052] Cooperative queue tracking error
[0053]
[0054] in, Indicates the tracking error of the position. and It is an intermediate variable, d i (t)=x i-1 (t)-x i (t) is the adjacent distance between agents. This is the safe distance when stationary, where e represents the natural constant. v0(t) represents the headway of the vehicle, and v0(t) represents the speed of the navigator. The tracking error represents the speed. β represents the limiting value of t as it approaches 0. i It is a constant greater than zero.
[0055] Step 3: Establish a network of reinforcement learning commentators:
[0056] Reinforcement Learning Commentator Network
[0057]
[0058] in, Represents the cost function. Let τ represent the integral from time 0 to positive infinity, where τ is the variable of integration. Represents the discount factor. u(t)=[u1(t),u2(t),...,u i (t)] T , and It is a positive definite matrix.
[0059]
[0060] in, ξ represents the estimated value of the cost function. c (e x ) is the activation function of the commentator network, W c These are the ideal weights for the commentator's neural network. It is W c The estimated value, ε c It is the approximation error of the commentator's neural network.
[0061] Commentator Neural Network Update Law
[0062]
[0063] in, The learning rate represents the commentator network.
[0064] Step 4: Establish a reinforcement learning actor network:
[0065] Reinforcement learning actor network
[0066]
[0067] in, W is the activation function of the actor network. ai These are the ideal weights for the actor's neural network. and They are With W ai The estimated value, ε ai It is the approximation error of the actor's neural network.
[0068] Actor Neural Network Update Law
[0069]
[0070] in, Represents the dynamic learning rate. κ ai ε1 and ε2 are both constants greater than zero. It is the expected cumulative cost. It is a constant greater than zero.
[0071] Step 5: Establish the queue integration sliding surface:
[0072] Integral sliding surface
[0073]
[0074] Among them, s i (t) is the sliding surface. Let δ, λ1, and λ2 represent the integral from time 0 to time t, where δ, λ1, and λ2 are all constants greater than zero.
[0075]
[0076] Among them, S i (t) is the coupled sliding surface, 0 < q ≤ 1, and N is the number of agents.
[0077] Step 6: Establish an adaptive integral sliding mode cooperative queue control strategy based on reinforcement learning:
[0078] Adaptive control parameter update law
[0079]
[0080] Where, γ ai γ ui , σ ai and σ ui It is a constant greater than zero. and They are ε ai With ε ui The estimated value.
[0081] Cooperative queue control law
[0082]
[0083] Where ρ, η, ζ, α, and k are all positive constants, || is the absolute value, and tanh() is the hyperbolic tangent function.
[0084] Effects of the embodiments of the present invention:
[0085] The beneficial effect of this invention lies in its effective solution to the problem that traditional cooperative queue control methods typically only consider queue stability and tracking error accuracy in their design, while neglecting energy efficiency, a key performance indicator, and often failing to ensure minimal energy consumption. This invention ensures the internal stability and overall queue stability of the entire queue, while achieving optimal control performance, improving queue efficiency, and enhancing queue security.
Claims
1. A reinforcement learning based adaptive integral sliding mode cooperative platoon control strategy, characterized in that The reinforcement learning-based critic-actor network architecture executes a control policy through an actor network, evaluates the pros and cons of the policy through a critic network, and adjusts neural network weights online through reinforcement learning to achieve stability and optimal control performance of the entire platoon. The control policy comprises the following steps: Step one, establishing a platoon system model with uncertainty, external disturbance and actuator saturation; Step two, establishing a cooperative platoon tracking error; Step three, establishing a reinforcement learning critic network; Step four, establishing a reinforcement learning actor network; Step five, establishing a platoon integral sliding mode surface; Step six, establishing a reinforcement learning-based adaptive integral sliding mode cooperative platoon control strategy; In the step one, platoon system model where i is the index, x i (t) and v i (t) represent the position and velocity of the ith agent, respectively, and t denotes time, represent column vectors containing position, velocity, and time, is the uncertainty and external disturbance to the system, and u i (t) is the control input, 0 < θ i ≤ 1 represents the degree of actuator saturation, and ε ui (t) denotes the saturation function approximation error. In the step two, cooperative platoon tracking error wherein is a tracking error of position, and is an intermediate variable, d i (t) = x i-1 (t) - x i (t) is the inter-agent distance of the agent, is a safety distance at rest, e represents a natural constant, denotes the headway, v0(t) denotes the speed of the leader, is a tracking error of speed, denotes the limit value for t tending to 0, β i is a constant greater than zero. In the step three, reinforcement learning critic network wherein represents a cost function, denotes an integral from time 0 to positive infinity, τ is the integration variable, represents a discount factor, u(t) = [u1(t), u2(t),..., u i (t)] T , and are positive definite matrices. where, represents an estimate of the cost function, ξ c (e x ) is an activation function of the critic network, W c is an ideal weight of the critic neural network, is an estimate of W c , ε c is an approximation error of the critic neural network. critic neural network update law wherein, the learning rate representative of the critic network, In the step four, reinforcement learning actor network wherein, is an activation function of the actor network, W ai is an ideal weight of the actor neural network, and are, respectively, and W ai is an estimate of the actor neural network, ε ai is an approximation error of the actor neural network. actor neural network update law wherein, represents a dynamic learning rate, κ ai , ∈1 and ∈2 are constants greater than zero, is a desired cumulative cost, is a constant greater than zero. In the step five, integral sliding mode surface where s i (t) is a sliding surface, denotes the integral from time 0 to time t, and δ, λ1and λ2are constants greater than zero. where S t (t) is the coupled sliding surface, 0 < q < 1, and N is the number of agents. In the step six, adaptive control parameter update law where γ ai , γ ui , σ ai and σ ui are constants greater than zero, and are estimates of ε ai and ε ui , respectively. cooperative platoon control law wherein ρ, η, ζ, α and k are normal numbers, || is an absolute value, and tanh() is a hyperbolic tangent function,
