Nonlinear multi-agent system consistency method based on reinforcement learning sliding mode control

Abstract

The invention discloses a non-linear multi-agent system consistency method based on reinforcement learning sliding mode control, and the method comprises the steps: initializing the states of all agents and the state of a leader, setting the weights of a reviewer network and an actor network, and introducing the weights into a virtual controller; a virtual controller is introduced to process communication delay, a distributed fault observer is designed to observe unknown faults of the system, an observed value is used for feedback compensation, meanwhile, a sliding mode controller is designed to enhance robustness of the system and suppress input saturation, and a reinforcement learning technology is combined to optimize a control strategy of each agent. And the state consistency can be realized in an approximately optimal mode. According to the method, the negative effects of system faults, input saturation and communication delay can be effectively eliminated, it is ensured that all agents can finally follow the leader, and optimal consistency is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of consistency control of multi-agent systems, and specifically to a consistency method for nonlinear multi-agent systems based on reinforcement learning sliding mode control. Background Art

[0002] In the study of multi-agent systems (MAS), achieving consistent control of the system is an important and challenging task. Traditional control methods often have difficulty coping with nonlinear systems and their complexity in practical applications. A multi-agent system consists of multiple interacting agents that communicate and cooperate to complete specific tasks. In the real world, many systems exhibit nonlinear characteristics, making the control problem more complex and difficult to handle. The characteristic of a nonlinear system is that the relationship between its state variables is no longer linear, which makes traditional linear control methods ineffective or even ineffective.

[0003] In the existing technology, in order to solve the consistency problem of nonlinear multi-agent systems, commonly used methods include but are not limited to: 1) Traditional sliding mode control, by designing the sliding surface, the system state slides along the sliding surface and eventually tends to the equilibrium point, thereby enhancing the robustness and anti-interference ability of the system; 2) Adaptive control, by adjusting the control parameters online to adapt to the changes and uncertainties of the system; 3) Model predictive control, by predicting the system behavior in the future and optimizing the current control strategy accordingly; 4) Cluster control, using swarm intelligence algorithms to coordinate the behavior of multiple agents to achieve the overall goal.

[0004] Although the above methods can solve the consistency problem of nonlinear multi-agent systems to a certain extent, there are still some shortcomings. For example, although traditional sliding mode control is robust, it is easy to produce chattering during the design and implementation process, which affects the system performance; adaptive control requires a lot of computing resources to adjust parameters in real time, which increases the burden on the system; model predictive control relies on accurate system models, but it is often difficult to obtain accurate models in practical applications; cluster control may fall into local optimal solutions due to the lack of a global perspective. Therefore, how to effectively eliminate the negative effects of system failures, input saturation and communication delays, and ensure that all agents can eventually follow the leader and achieve optimal consistency, has become a key issue that needs to be solved urgently. Summary of the invention

[0005] In order to effectively eliminate the negative impact of system failures, input saturation and communication delays, ensure that all agents can eventually follow the leader and achieve optimal consistency control of the multi-agent system, the present application provides a nonlinear multi-agent system consistency method based on reinforcement learning sliding mode control.

[0006] In a first aspect, the present application provides a nonlinear multi-agent system consistency method based on reinforcement learning sliding mode control, comprising: S1. Initialize the control protocol, including: initializing the states of all agents and the leader, introducing a virtual controller, and initializing the weights of the critic network and the actor network; S2. Design a fault observer to observe and estimate the nonlinear fault terms in the multi-agent system and use the observed values ​​for the feedback control protocol; S3, design the optimal controller, input the current state of the multi-agent into the actor network, obtain the preliminary control protocol, and adjust the control protocol according to the feedback of the critic network; S4. Design a sliding mode controller, define a sliding surface for each agent, calculate the sliding surface according to the current state of the agent, and design a sliding mode controller for each agent; S5. Combining sliding mode control and reinforcement learning control to build a composite control protocol; S6, applying the control protocol, including: calculating the virtual controller and the control protocol and applying them to the multi-agent system, driving the multi-agent to move according to the designed trajectory; collecting the operation data of the multi-agent system in real time, performing performance analysis on the operation data and optimizing the control strategy based on the performance analysis results; S7, updating the neural network weights of the critic network and the actor network; including: updating the weights of the critic network and the actor network using a reinforcement learning algorithm during the iteration process of the control strategy of the multi-agent system according to the collected operation data of the multi-agent system; S8. Perform strategy iteration, check convergence, and output the final control protocol; including: using the strategy iteration method of reinforcement learning to optimize the control strategy of each agent; when the multi-agent system reaches the predetermined convergence conditions, output the final control strategy.

[0007] By adopting the above scheme, a virtual controller is introduced to deal with communication delays, a distributed fault observer is designed to observe unknown faults in the system, and the observation values ​​are used for feedback compensation. At the same time, a sliding mode controller is designed to enhance the robustness of the system and suppress input saturation. Reinforcement learning technology is combined to optimize the control strategy of each intelligent agent, so that it can achieve state consistency in a near-optimal way.

[0008] Preferably, the initialization control protocol in step S1 specifically includes: S11, initialize the agent state; each agent The dynamic model is expressed as: In the formula, , is a positive integer, Representing an Agent No. Stage status, is a local nonlinear function, For intelligent agents The control input, Representing an Agent Bounded uncertainty; set the initial state of all agents ; S12, initialize the leader state; the dynamic model of the leader is expressed as: , where Indicates the leader’s Stage status, is the local nonlinear function of the leader, Indicates bounded interference of the leader; sets the initial state of the leader ; S13, establish a local neighborhood error dynamic model; including: define the local neighborhood error It is expressed as: In the formula, express Adjacency Matrix Agent and Agent The connection relationship and its weight, is the number of agents. If and Agent If there is a connection between ; If the agent and Agent If there is no connection between ; For Node If there is a fixed gain from the leader to the node The edge of ,otherwise ; Combining the dynamic model of the agent and the definition of local neighborhood error, the dynamic model of local neighborhood error is obtained: In the formula, is a local nonlinear function, For intelligent agents The control input, Representing an Agent bounded uncertainty; S14. In order to consider the communication delay, a virtual controller is introduced: , so that the local neighborhood error dynamic model is expressed as: ; To handle the communication delay problem, the virtual controller is represented as: In the formula, The control outputs for the optimal controller and integral sliding mode controller constructed based on the subsequent design; It is assumed that The communication delay is bounded by is a positive integer, set ; S15. Initialized critic and actor weights, including setting the initial weight values ​​of the critic network and actor network in reinforcement learning.

[0009] By adopting the above scheme, we can ensure that the initial conditions of all agents and leaders are consistent, avoid control instability or failure caused by different initial conditions, establish a local neighborhood error dynamic model to monitor and adjust the state of each agent in real time, introduce a virtual controller to predict the future state, compensate in advance, effectively deal with the communication delay problem, and ensure the stability and consistency of the multi-agent system.

[0010] Preferably, the design structure of the fault observer in step S2 is: In the formula, is the fault observer state, is an unknown uncertainty, is the output of the fault observer, is the known matrix of the observer.

[0011] By adopting the above scheme, unknown uncertainties are taken into account, the nonlinear fault terms in the system are observed and estimated, and the observed values ​​are used in the feedback control protocol, which effectively eliminates the impact of system failures and improves the consistency control performance of the multi-agent system.

[0012] Preferably, designing the optimal controller in step S3 specifically includes: S31. For any stable control strategy , define the value function It is expressed as: In the formula, is a positive semidefinite matrix used to measure the weight of the state deviation of the multi-agent system; is a positive matrix used to measure the weight of the control input; is the discount factor; S32. Based on optimal control and dynamic programming theory, the Hamiltonian function is proposed and defined as: In the formula, is the communication delay coefficient; S33, the optimal control protocol is: In the formula, are known control parameters to be determined, is an adaptive term, The specific definition is as follows: ; S34, using reinforcement learning method to solve HJB equation; S35, Policy Iteration RL Based on Actor-Critic Neural Network; For Agents , the value function is usually approximated by a neural network as: In the formula, is the neural network activation function vector, Neural network approximation error; estimated value of the hypothesized value function and the estimated value of the optimal control protocol for: In the formula, Both represent weight matrices, for The amount of change, To control the parameters, for The gradient operator.

[0013] By adopting the above scheme, the value function is defined, the Hamiltonian function is constructed, the control strategy is evaluated, the optimal control protocol is designed under the joint action of known unknown control parameters and adaptive terms, the reinforcement learning method is used to approximate the solution of the HJB equation, and the control strategy is refined and optimized through policy iteration RL based on the actor-critic neural network to ensure that all intelligent agents ultimately achieve global consistent control.

[0014] Preferably, designing the sliding mode controller in step S4 specifically includes: S41, for intelligent agents The integral sliding surface is constructed as: In the formula, yes Function of is a constant, ; A is defined as follows: ; S42. For intelligent agents , sliding mode controller Designed for: In the formula, is an adaptive term, which is a known constant upper bound; S43, the adaptive item The derivative of is defined as: In the formula, is a constant; definition: In the formula, is a normal number.

[0015] By adopting the above scheme, an integral sliding surface is constructed for each intelligent agent, and a sliding mode controller is designed to effectively deal with the uncertainty and input saturation problems of the system. At the same time, adaptive terms are introduced and derivatives are designed for adaptation to further optimize the adaptive mechanism and improve the adaptive ability and robustness of the system.

[0016] The combination of sliding mode control and reinforcement learning control in step S5 to construct a composite control protocol is specifically expressed as follows: In the formula, is the fault observer state, is the known input saturation suppression parameter, is the known delay compensation coefficient, is the optimal controller calculated in step S3, is the agent in step S4 The sliding mode controller.

[0017] By adopting the above scheme, the observation results of the fault observer, input saturation, communication delay and other methodological issues are taken into consideration, and the sliding mode control and reinforcement learning control are effectively integrated. This not only improves the control accuracy of a single agent, but also ensures that all agents can eventually follow the leader and achieve state consistency on a global scale.

[0018] Preferably, the application of the control protocol and the real-time collection of the operation data of the multi-agent system in step S6 specifically include: S61, Computing Virtual Controller : ,in, ; S62, calculation control protocol, control input Calculated by the following formula: In the formula, For virtual output, It is status feedback; S63, applying the calculated control protocol to the multi-agent system to drive the agents to move according to the designed trajectory; S64, during the movement of the multi-agent system, real-time collection of multi-agent system operation data, including: state information and control signals; S65. Optimizing the control strategy, including: analyzing the operation data results, evaluating the performance index values ​​under different control strategies according to the preset performance evaluation index, adjusting the derivative of the adaptive term in the sliding mode controller or the network parameters of the critic network or the actor network to complete the optimization of the control strategy.

[0019] By adopting the above scheme, the virtual controller not only takes into account the contribution of the optimal controller, but also integrates the sliding mode controller's ability to suppress input saturation, effectively solves communication delays, and obtains more accurate predicted outputs; combining the virtual controller output and state feedback, the control input required for each agent is calculated and applied to the multi-agent system, thereby improving the flexibility and robustness of the control strategy.

[0020] Preferably, the neural network weights for the critic network and the actor network in step S7 specifically include: S71. For each agent, the estimate of the Hamiltonian function is: In the formula, It’s an immediate return. is the discount factor, is the gradient of the state feature, A and B are the system matrices, is the coefficient, is the adjacency matrix element, It's a delay. is the residual error; S72, minimize the residual error, design weights and To minimize the squared residual error; ; S73. Update the commentator network weight, the formula is: ; further, In the formula, In the formula, is the learning rate of the critic network; Used for normalization; S74. Update the actor network weight update, the formula is: In the formula, , , , and is the tuning parameter, is the learning rate of the actor network.

[0021] By adopting the above scheme, accurate Hamiltonian function estimates are generated by combining the instantaneous rewards, value function estimates, and state feature gradients of each agent in the actual operation process; the weight sum is designed to minimize the squared residual error to ensure that the direction of weight update is always moving towards the goal of reducing the error; the combination of accurate Hamiltonian function estimation and efficient weight update mechanism improves the consistency control performance of the multi-agent system and enhances the robustness and response speed of the system.

[0022] Preferably, performing strategy iteration, checking convergence, and outputting the final control protocol in step S8 specifically includes: S81. Calculate the reward value of each agent under the current control strategy and evaluate the current control strategy; use the policy iteration method in reinforcement learning to update the control strategy of each agent so that it gradually approaches the optimal control strategy; S82, setting a convergence condition, after each control strategy iteration, checking whether the current control strategy satisfies the convergence condition; if so, it is considered that the control strategy has converged; otherwise, returning to continue the control strategy iteration; S83. When the control strategy converges, determine the final control strategy.

[0023] By adopting the above scheme, based on strategy iteration and optimization as well as the preset and check of convergence conditions, the optimal control strategy is finally obtained.

[0024] In summary, this application has the following beneficial effects: By introducing a virtual controller and designing a distributed fault observer, the impact of communication delays and unknown system faults is effectively dealt with, thereby improving the robustness and reliability of the system. By combining sliding mode control with reinforcement learning technology, a composite control protocol is formed, which not only enhances the system's anti-interference ability, but also effectively suppresses input saturation, thereby improving the system's control accuracy and response speed. The reinforcement learning algorithm is used to continuously optimize the control strategy of each intelligent agent, so that all intelligent agents can achieve optimal consistency control on a global scale, ensuring that the system performance reaches the best optimization goal. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of the method described in a specific embodiment; Figure 2 A flowchart for running the method described in the specific embodiment; Figure 3 is a specific flow chart of step S1 in the method described in the specific embodiment; Figure 4 is a specific flow chart of step S3 in the method described in the specific embodiment; Figure 5 is a specific flow chart of step S4 in the method described in the specific embodiment; Figure 6 is a specific flow chart of step S6 in the method described in the specific embodiment; Figure 7 is a specific flow chart of step S7 in the method described in the specific embodiment; Figure 8 It is a specific flow chart of step S8 in the method described in the specific embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] like Figure 1 As shown, the embodiment of the present application discloses a nonlinear multi-agent system consistency method based on reinforcement learning sliding mode control, which specifically includes: S1. Initialize the control protocol.

[0028] The states of all agents and leaders are accurately initialized, and the initial weights of the critic network and the actor network are set to ensure that the states of all agents and leaders are well initialized, laying the foundation for subsequent control strategies. Figure 3 As shown, the specific steps of the initialization control protocol include: S11. Initialize the agent state.

[0029] For each agent Construct a dynamic model, which is specifically expressed as follows: In the formula, , is a positive integer, Representing an Agent No. Stage status, is a local nonlinear function, For intelligent agents The control input, Representing an Agent Bounded uncertainty; set the initial state of all agents .

[0030] Considering nonlinear functions plays an important role in many applications, especially in the analysis and control of nonlinear systems. Assume that there is a damping system of an intelligent agent (drone), whose dynamic equation is: ,in, is the state vector, is the control input, and is a nonlinear function; if Satisfies the nonlinear condition, that is, there is a constant Make , for all and , a state feedback controller can be designed The stability of the closed-loop system is guaranteed.

[0031] S12. Initialize the leader state.

[0032] A dynamic model is constructed for leaders, which is specifically expressed as follows: , where Indicates the leader’s Stage status, is the local nonlinear function of the leader, Indicates bounded interference of the leader; sets the initial state of the leader .

[0033] S13. Establish a local neighborhood error dynamic model.

[0034] In order to accurately describe the relative position and speed differences between each agent and its neighbors, the state of each agent can be monitored and adjusted in real time to enhance the coordination and consistency of the multi-agent system. Specifically, a local neighborhood error dynamic model is established. First, the local neighborhood error is defined. It is expressed as: In the formula, express Adjacency Matrix Agent and Agent The connection relationship and its weight, is the number of agents. If and Agent If there is a connection between ; If the agent and Agent If there is no connection between ; For Node If there is a fixed gain from the leader to the node The edge of ,otherwise ; Combining the dynamic model of the agent and the definition of local neighborhood error, the dynamic model of local neighborhood error is obtained: In the formula, is a local nonlinear function, For intelligent agents The control input, Representing an Agent of bounded uncertainty.

[0035] S14. Introduce a virtual controller.

[0036] In order to effectively handle the communication delay problem so that the system can maintain stable operation even in the case of network delay, a virtual controller is defined: , and then, the local neighborhood error dynamic model is expressed as: , so that the local neighborhood error dynamic model is expressed as: ; Considering the actual communication delay problem, the virtual controller is expressed as: ; In the formula, The control outputs for the optimal controller and integral sliding mode controller constructed based on the subsequent design; It is assumed that The communication delay is bounded by is a positive integer, set .

[0037] S15. Initialized critic and actor weights, including setting the initial weight values ​​of the critic network and actor network in reinforcement learning.

[0038] It provides a reliable starting point for subsequent strategy iteration and optimization, including: setting the initial weight values ​​of the critic network and actor network in reinforcement learning; such as: the initial weight value of the critic network Set to 1, the initial value of the weight of the actor network is 1.

[0039] S2. Design a fault observer.

[0040] Specifically, in order to effectively eliminate the impact of system failures on consistency control, a distributed fault observer is designed to observe and estimate unknown system failures, and use the observed values ​​for feedback compensation.

[0041] Specifically, the design structure of the fault observer is: In the formula, is the fault observer state, is an unknown uncertainty, is the output of the fault observer, is the known matrix of the observer.

[0042] S3. Design the optimal controller.

[0043] Specifically, the optimal controller is designed based on Hamiltonian function, optimal control protocol, reinforcement learning method and actor-critic neural network. The current state of multiple agents is input into the actor network to obtain a preliminary control protocol. The control protocol is adjusted according to the feedback of the critic network to optimize the control strategy, so as to ensure that all agents can eventually achieve global consistency control and achieve optimal control. Figure 4 As shown, the specific steps include: S31. Define the value function.

[0044] For any stable control strategy , define the value function It is expressed as: In the formula, is a positive semidefinite matrix used to measure the weight of the state deviation of the multi-agent system, and can have a zero eigenvalue, that is, there is a non-zero vector Make ; is a positive matrix used to measure the weight of the control input, and all eigenvalues ​​are positive; is the discount factor, which can be any value.

[0045] S32. Define Hamiltonian function.

[0046] Based on optimal control and dynamic programming theory, the Hamiltonian function is proposed and defined as: In the formula, is the communication delay coefficient, .

[0047] Define the optimal control protocol.

[0048] Specifically, under the joint action of known unknown control parameters and adaptive terms, the system state is effectively regulated to reduce the impact of system uncertainty. The optimal control protocol is defined as: In the formula, are known control parameters to be determined, is an adaptive term, The specific definition is as follows: .

[0049] S34. Solve the HJB equation using reinforcement learning method.

[0050] In order to obtain a near-optimal control protocol, a reinforcement learning approach is used to solve the HJB equations, since directly solving the HJB equations is challenging in deriving analytical solutions.

[0051] S35. Policy iteration RL based on actor-critic neural network.

[0052] Specifically, for the intelligent agent , the value function is usually approximated by a neural network as: In the formula, is the neural network activation function vector, Neural network approximation error; estimated value of the hypothesized value function and the estimated value of the optimal control protocol for: In the formula, Both represent weight matrices, for The amount of change, To control the parameters, for The gradient operator.

[0053] S4. Design a sliding mode controller.

[0054] In order to effectively deal with the uncertainty and input saturation of the system, a sliding surface is defined for each agent, the sliding surface is calculated according to the current state, and the sliding mode controller of each agent is designed. Figure 5 As shown, the specific steps include: S41. Construct an integral sliding surface.

[0055] For intelligent agents The integral sliding surface is constructed as: In the formula, yes Function of is a constant, ; A is defined as follows: .

[0056] S42. Design a sliding mode controller.

[0057] For the intelligent agent , sliding mode controller Designed for: In the formula, is an adaptive term and is a known constant upper bound.

[0058] S43. Define the derivative of the adaptive term.

[0059] The adaptive item The derivative of is defined as: In the formula, is a constant; definition: In the formula, for The deviation variable, is a normal number.

[0060] In order to solve the calculation problem of the sliding mode controller, it is necessary to obtain :Due to communication delay, this information cannot be obtained at time t. In practice, this is usually very small and can be used replace Perform calculations.

[0061] S5. Combine sliding mode control and reinforcement learning control to build a composite control protocol. Specifically expressed as:

[0062] In the formula, is the fault observer state, is the known input saturation suppression parameter, is the known delay compensation coefficient, is the optimal controller calculated in step S3, is the agent in step S4 The sliding mode controller.

[0063] Application Control Protocol.

[0064] like Figure 6 As shown, the specific steps include: S61, calculating the virtual controller. Specifically, the virtual controller calculates the virtual controller : .

[0065] S62, calculation control protocol. Control input Calculated by the following formula: In the formula, For virtual output, It is status feedback.

[0066] S63, multi-agent system application control protocol. Figure 2 As shown, the calculated control protocol is applied to the multi-agent system to drive the agents to move according to the designed trajectory.

[0067] S64. Real-time collection of multi-agent system operation data. During the movement of the multi-agent system, real-time collection of multi-agent system operation data is performed, including: status information and control signals. The collected data is analyzed to identify system performance bottlenecks and improvement directions.

[0068] S65. Optimize control strategy. Based on the data analysis results, adjust and optimize the control strategy to improve the robustness and response speed of the system.

[0069] Specifically, it includes: analysis of running data, such as: tracking error analysis: by observing the tracking error trajectory between the agent state and the leader state, we can understand the consistency of the system under different control strategies. A smaller tracking error indicates that the system is closer to the consistency state. If the error is large or has obvious fluctuations, it means that there is a problem with the system; or analysis of performance indicators: pre-define performance indicators, and quantitatively evaluate the impact of control strategies on system performance by calculating and comparing performance indicator values ​​under different control strategies; these indicators can reflect the comprehensive performance of the system in the control process, such as energy consumption, control accuracy, etc. When analyzing data, paying attention to the size and change trend of these performance indicators will help judge the pros and cons of the control strategy. Another example: for neural network weight analysis: in the policy iteration RL based on Actor-Critic neural network, the change of neural network weight reflects the progress of the learning process and the adjustment of the control strategy. By observing whether the weights converge and the speed and stability of convergence, we can understand the learning effect of the RL algorithm on the control strategy. If the weights can converge to appropriate values ​​quickly and stably, it means that the RL algorithm can effectively learn a better control strategy; on the contrary, if the weights fluctuate greatly or fail to converge, it may be necessary to adjust the learning rate, network structure or other related parameters; Then, through the above analysis, the control strategy is adjusted and optimized, including: parameter adjustment, such as: in the sliding mode controller, the derivative of the adaptive term can be adjusted according to the state and error of the sliding surface in the system operation data The value of can be adjusted to optimize the adaptive process and enable the controller to better cope with uncertainty. For the communication delay coefficient, if the system is found to be sensitive to communication delay in data analysis, the value of can be appropriately adjusted to improve the control performance. For example, if the tracking error increases significantly when the communication delay is large, you can try to reduce it to enhance the compensation effect for the delay. Alternatively, the method of optimizing the structure and parameters of the neural network can be adopted. For the Actor-Critic neural network, the network structure can be adjusted according to the results of data analysis, such as increasing the number of neurons in the hidden layer and changing the activation function. If it is found that the network converges too slowly or overfits during the learning process, the number of neurons can be increased or a more appropriate regularization method can be adopted. At the same time, optimize parameters such as the learning rate, observe the impact on the learning effect of the control strategy, and select a parameter combination that can make the network converge quickly and stably and have good performance. For example, if oscillation is found during the weight update process, it may be that the learning rate is too large and the learning rate needs to be appropriately reduced.

[0070] S7. Update the neural network weights of the critic network and the actor network.

[0071] In order to further improve the consistency control performance of the multi-agent system and enhance the robustness and response speed of the system, based on the collected operation data of the multi-agent system, in the control strategy iteration process of the multi-agent, accurate Hamiltonian function estimation and efficient weight update mechanism are performed according to the runtime data to effectively improve the weight update efficiency of the critic network and the actor network, thereby optimizing the control strategy of the multi-agent system.

[0072] like Figure 7 As shown, the specific steps include: S71. Hamiltonian function estimation. For each agent, the Hamiltonian function is estimated as: In the formula, It’s an immediate return. is the discount factor, is the gradient of the state feature, A and B are the system matrices, is the coefficient, is the adjacency matrix element, It's a delay. is the residual error.

[0073] S72, minimize the squared residual error. Specifically, design the weights and To minimize the squared residual error; .

[0074] S73, update the weight of the critic network. Through the normalized gradient descent algorithm, the weight update formula of the critic network is: ; Furthermore, In the formula, In the formula, is the learning rate of the critic network, Used for normalization.

[0075] S74. Update the actor network weight update, the formula is: In the formula, , , , and is the tuning parameter, is the learning rate of the actor network.

[0076] At the same time, the state of the agent is adjusted to ensure the continuous optimization of the control strategy. The adjustment of the agent state is achieved through the control input, which is closely related to the control protocol. The following is the specific process: Agent Control input Depend on Calculated, among which is the optimal controller part learned by RL algorithm, It is the sliding mode controller part; in each iteration, according to the current state of the agent Neighbor agent information (via embodied) and the learned controller parameters to calculate the control input ; Complete the dynamic update of the agent state, including: after getting the control input After that, it is substituted into the state equation and the state of the agent is updated by numerical integration and other methods; during the simulation process, the state is updated according to a certain time step. As time goes by, the state of the agent changes continuously, and under the action of the control input, it develops in the direction consistent with the leader's state to achieve the consistency goal; combined with the continuous adjustment of the learning process: during the learning process of the RL algorithm, as the weights of the neural network are updated, the control protocol and is also being optimized; this results in the control input , and then continuously adjust the state of the agent; for example, when the Critic neural network adjusts the weight according to the system's operating data (such as tracking error, performance indicators, etc.), it will change the estimate of the value function, thereby affecting the optimal controller The update of the Actor neural network weights will directly change the choice of control strategy, allowing the agent to take better actions in subsequent state updates to better cope with system uncertainties and achieve optimal consistency.

[0077] S8. Perform strategy iteration, check convergence, and output the final control protocol.

[0078] In order to improve the consistency control performance of nonlinear multi-agent systems, the policy iteration method of reinforcement learning is used to optimize the control strategy of each agent so that it gradually approaches the optimal control protocol, thereby achieving consistent control on a global scale; when the multi-agent system reaches the predetermined convergence condition, the final control strategy is output to ensure that all agents reach a consistent state and the system performance reaches the optimization goal. Figure 8 As shown, the specific steps include: S81. Strategy iteration.

[0079] Calculate the reward value of each agent under the current control strategy and evaluate the current control strategy; use the policy iteration method in reinforcement learning to update the control strategy of each agent so that it gradually approaches the optimal control strategy; S82. Check convergence.

[0080] Set the convergence condition. After each control strategy iteration, check whether the current control strategy meets the convergence condition. If so, the control strategy is considered to have converged. Otherwise, return to continue the control strategy iteration.

[0081] S83, output the final control protocol. When the strategy converges, determine the final optimal control strategy. Use the optimal control strategy to make all agents achieve global consistency control. Ensure that all agents achieve consistency in the global scope and make the system performance reach the optimization goal.

[0082] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A nonlinear multi-agent system consensus method based on reinforcement learning sliding mode control, characterized in that: include: S1. Initialize the control protocol, including: initializing the states of all agents and the leader, introducing a virtual controller, and initializing the weights of the critic network and the actor network; S2. Design a fault observer to observe and estimate the nonlinear fault terms in the multi-agent system and use the observed values ​​for the feedback control protocol; S3, design the optimal controller, input the current state of the multi-agent into the actor network, obtain the preliminary control protocol, and adjust the control protocol according to the feedback of the critic network; S4. Design a sliding mode controller, define a sliding surface for each agent, calculate the sliding surface according to the current state of the agent, and design a sliding mode controller for each agent; S5. Combining sliding mode control and reinforcement learning control to build a composite control protocol; S6, applying the control protocol, including: calculating the virtual controller and the control protocol and applying them to the multi-agent system, driving the multi-agent to move according to the designed trajectory; collecting the operation data of the multi-agent system in real time, performing performance analysis on the operation data and optimizing the control strategy based on the performance analysis results; S7, updating the neural network weights of the critic network and the actor network; including: updating the weights of the critic network and the actor network using a reinforcement learning algorithm during the iteration process of the control strategy of the multi-agent system according to the collected operation data of the multi-agent system; S8. Perform strategy iteration, check convergence, and output the final control protocol; including: using the strategy iteration method of reinforcement learning to optimize the control strategy of each agent; when the multi-agent system reaches the predetermined convergence conditions, output the final control strategy.

2. The nonlinear multi-agent system consistency method based on reinforcement learning sliding mode control according to claim 1 is characterized in that: Initializing the control protocol in step S1 specifically includes: S11, initialize the agent state; each agent The dynamic model is expressed as: In the formula, , is a positive integer, Representing an Agent No. Stage status, is a local nonlinear function, For intelligent agents The control input, Representing an Agent Bounded uncertainty; set the initial state of all agents ; S12, initialize the leader state; the dynamic model of the leader is expressed as: , where Indicates the leader’s Stage status, is the local nonlinear function of the leader, Indicates bounded interference of the leader; sets the initial state of the leader ; S13, establish a local neighborhood error dynamic model; including: define the local neighborhood error It is expressed as: In the formula, express Adjacency Matrix Agent and Agent The connection relationship and its weight, is the number of agents. If and Agent If there is a connection between ; If the agent and Agent If there is no connection between ; For Node If there is a fixed gain from the leader to the node The edge of ,otherwise ; Combining the dynamic model of the agent and the definition of local neighborhood error, the dynamic model of local neighborhood error is obtained: In the formula, is a local nonlinear function, For intelligent agents The control input, Representing an Agent bounded uncertainty; S14. In order to consider the communication delay, a virtual controller is introduced: , so that the local neighborhood error dynamic model is expressed as: ; To handle the communication delay problem, the virtual controller is represented as: In the formula, The control outputs for the optimal controller and integral sliding mode controller constructed based on the subsequent design; It is assumed that The communication delay is bounded by is a positive integer, set ; S15. Initialized critic and actor weights, including setting the initial weight values ​​of the critic network and actor network in reinforcement learning.

3. The nonlinear multi-agent system consistency method based on reinforcement learning sliding mode control according to claim 1 is characterized in that: The design structure of the fault observer in step S2 is: In the formula, is the fault observer state, is an unknown uncertainty, is the output of the fault observer, is the known matrix of the observer.

4. The nonlinear multi-agent system consistency method based on reinforcement learning sliding mode control according to claim 1, characterized in that: Designing the optimal controller in step S3 specifically includes: S31. For any stable control strategy , define the value function It is expressed as: In the formula, is a positive semidefinite matrix used to measure the weight of the state deviation of the multi-agent system; is a positive matrix used to measure the weight of the control input; is the discount factor; S32. Based on optimal control and dynamic programming theory, the Hamiltonian function is proposed and defined as: In the formula, is the communication delay coefficient; S33, the optimal control protocol is: In the formula, are known control parameters to be determined, is an adaptive term, The specific definition is as follows: ; S34, using reinforcement learning method to solve HJB equation; S35, Policy Iteration RL Based on Actor-Critic Neural Network; For Agents , the value function is usually approximated by a neural network as: In the formula, is the neural network activation function vector, Neural network approximation error; estimated value of the hypothesized value function and the estimated value of the optimal control protocol for: In the formula, Both represent weight matrices, for The amount of change, To control the parameters, for The gradient operator.

5. The nonlinear multi-agent system consistency method based on reinforcement learning sliding mode control according to claim 1, characterized in that: Designing the sliding mode controller in step S4 specifically includes: S41, for intelligent agents The integral sliding surface is constructed as: In the formula, yes Function of is a constant, ; A is defined as follows: ; S42. For intelligent agents , sliding mode controller Designed for: In the formula, is an adaptive term, which is a known constant upper bound; S43, the adaptive item The derivative of is defined as: In the formula, is a constant.

6. The nonlinear multi-agent system consistency method based on reinforcement learning sliding mode control according to claim 1, characterized in that: The combination of sliding mode control and reinforcement learning control in step S5 to construct a composite control protocol is specifically expressed as follows: In the formula, is the fault observer state, is a known input saturation suppression parameter, is the known delay compensation coefficient, is the optimal controller calculated in step S3, is the agent in step S4 The sliding mode controller.

7. The nonlinear multi-agent system consistency method based on reinforcement learning sliding mode control according to claim 1 is characterized in that: The application of the control protocol and real-time collection of operating data of the multi-agent system in step S6 specifically include: S61, Computing Virtual Controller : ; S62, calculation control protocol, control input Calculated by the following formula: In the formula, For virtual output, It is status feedback; S63, applying the calculated control protocol to the multi-agent system to drive the agents to move according to the designed trajectory; S64, during the movement of the multi-agent system, real-time collection of multi-agent system operation data, including: state information and control signals; S65. Optimizing the control strategy, including: analyzing the operation data results, evaluating the performance index values ​​under different control strategies according to the preset performance evaluation index, adjusting the derivative of the adaptive term in the sliding mode controller or the network parameters of the critic network or the actor network to complete the optimization of the control strategy.

8. The nonlinear multi-agent system consistency method based on reinforcement learning sliding mode control according to claim 1, characterized in that: The neural network weights for the critic network and the actor network in step S7 specifically include: S71. For each agent, the estimate of the Hamiltonian function is: In the formula, It’s an immediate return. is the discount factor, is the gradient of the state feature, A and B are the system matrices, is the coefficient, is the adjacency matrix element, It's a delay. is the residual error; S72, minimize the residual error, design weights and To minimize the squared residual error; ； S73. Update the commentator network weight, the formula is: ; further, In the formula, In the formula, is the learning rate of the critic network; Used for normalization; S74. Update the actor network weight update, the formula is: In the formula, , , , and is the tuning parameter, is the learning rate of the actor network.

9. The nonlinear multi-agent system consistency method based on reinforcement learning sliding mode control according to claim 1, characterized in that: The step S8 of performing strategy iteration, checking convergence, and outputting the final control protocol specifically includes: S81. Calculate the reward value of each agent under the current control strategy and evaluate the current control strategy; use the policy iteration method in reinforcement learning to update the control strategy of each agent so that it gradually approaches the optimal control strategy; S82, setting a convergence condition, after each control strategy iteration, checking whether the current control strategy satisfies the convergence condition; if so, it is considered that the control strategy has converged; otherwise, returning to continue the control strategy iteration; S83. When the control strategy converges, determine the final control strategy.

Images