IQL collaborative control method for AI intelligent control system

Abstract

The present invention belongs to the technical field of multi-agent collaborative control, and specifically relates to an IQL collaborative control method of an AI intelligent control system, the steps of which include constructing a virtual controller based on dynamic asynchrony for a multi-agent system MAS composed of N agents; introducing a cancellation model with multiple interferences in the virtual controller; for the IQL multi-agent system, using a virtual controller to improve the action selection process of the IQL multi-agent system, making each agent equivalent to an interrelated intelligent system rather than an independent individual, thereby improving the control output effect of the IQL multi-agent system under environmental instability; based on the improved IQL multi-agent system, realizing collaborative control of N agents. The present invention improves the adaptability and robustness of the multi-agent system in an uncertain environment, and can maintain a stable collaborative control effect under complex interference conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-agent collaborative control, and specifically relates to an IQL collaborative control method for an AI intelligent control system. Background Art

[0002] Multi-Agent Systems (MAS) are systems composed of multiple autonomous agents that can collaborate, coordinate or compete with each other to complete complex tasks. Each agent has independent perception, decision-making and execution capabilities, and can operate flexibly in a distributed environment. Multi-agent systems are widely used in robot cluster control, multi-mine hoist systems, distributed computing and other fields. They can achieve tasks that are difficult for a single agent to complete through collaborative work, and perform well in handling dynamic and uncertain environments.

[0003] IQL (Independent Q-Learning) is an important algorithm in the field of reinforcement learning in artificial intelligence (AI), which is specifically used for independent learning and decision optimization in multi-agent systems. Through the reinforcement learning mechanism, IQL enables each agent to continuously adjust its own strategy through trial and error when interacting with the environment to maximize its long-term benefits without relying on the strategies of other agents. The relationship between IQL and AI is reflected in its help to AI systems solve the problems of cooperation and competition in multi-agent environments, especially in complex, dynamic, and distributed task scenarios. As part of AI technology, IQL provides AI systems with more flexible and adaptive solutions, enabling them to self-learn, optimize, and improve intelligent behavior in uncertain environments. It can be seen that IQL is a reinforcement learning algorithm in AI, and the main application scenario of IQL is multi-agent systems, so AI and MAS are strongly related.

[0004] With the increasing complexity of application fields, MAS generally suffers from environmental instability disturbances in multi-dimensional high-order nonlinear systems, which makes the coordinated control effect of the system under output constraints poor. To overcome the above defects, general solutions include the following distributed adaptive control and improved reinforcement learning intelligent control.

[0005] Distributed Adaptive Control: Distributed adaptive control allows agents to dynamically adjust their control parameters based on their own local information by running adaptive control algorithms independently in each agent. This approach enables each agent to adapt to changes in the environment and achieve global collaborative control by sharing local information with other agents. Although distributed adaptive control is able to cope with environmental changes, its response speed may be insufficient when dealing with complex nonlinear disturbances. In addition, due to its reliance on local information, this method leads to poor global coordination in extreme cases, especially when the system requires a high degree of synchronization.

[0006] Improved reinforcement learning intelligent control: Improved reinforcement learning intelligent control uses reinforcement learning algorithms to enable each agent to learn and optimize its control strategy in interaction with the environment. The agent gradually accumulates experience in long-term operation, identifies and responds to different environmental disturbances, and thus achieves better collaborative control effects. The main challenge of reinforcement learning is that the convergence speed is slow, especially in complex and changeable nonlinear environments, and the initial learning process is unstable. In addition, when there are output constraints, the policy optimization process of reinforcement learning is restricted, resulting in a decrease in global performance.

[0007] Therefore, in order to better optimize the multi-agent collaborative control strategy and overcome the constraints of environmental instability interference, it is imperative to study a new AI intelligent control system. Summary of the invention

[0008] In view of the above deficiencies in the prior art, the purpose of the present invention is to provide an IQL collaborative control method for an AI intelligent control system, which improves the adaptability and robustness of the multi-agent system in an uncertain environment and can maintain a stable collaborative control effect under complex interference conditions.

[0009] To achieve the above objectives, the present invention provides an IQL collaborative control method for an AI intelligent control system, comprising the following steps:

[0010] S1. For a multi-agent system MAS consisting of N agents, a virtual controller based on dynamic asynchrony is constructed;

[0011] S2. In the virtual controller, a cancellation model with multiple disturbances is introduced. The cancellation model is used to cancel the multiple external disturbances and internal coupling disturbances existing in the MAS, thereby stabilizing the coordinated control of the MAS.

[0012] S3. For the IQL multi-agent system, the virtual controller is used to improve the action selection process of the IQL multi-agent system, and each agent is equivalent to an interrelated intelligent system rather than an independent individual, thereby improving the control output effect of the IQL multi-agent system under environmental instability;

[0013] S4. Based on the improved IQL multi-agent system, collaborative control of N agents is achieved.

[0014] As a preferred solution of the present invention, in S1, the process of constructing a virtual controller based on dynamic asynchrony is:

[0015] S11. For N agents, the state of agent i at time t is represented by a vector , where i represents the index of the agent, , t represents time, Rn represents an n-dimensional real vector space; the control input of agent i at time t is , R m represents an m-dimensional real vector space, then the dynamic behavior of agent i at time t It is represented by the following state space equation:

[0016] (1);

[0017] In the formula, and are the system matrix and input matrix of MAS respectively, is the space consisting of real matrices with n rows and m columns ( Similarly, it is an n×n real matrix space); is the nonlinear disturbance term of agent i at time t, describing the uncertainty and nonlinear interference of the environment;

[0018] S12. Introducing virtual controller , and define it as:

[0019] (2);

[0020] In the formula, is the local feedback gain matrix; is the set of agents adjacent to agent i; is the coupling gain matrix between agent i and agent j; is the state vector of agent j at time t, and ;

[0021] S13, in order to improve the response speed of MAS in unstable environment, The asynchronous update mechanism is adopted in , that is, different agents can update their states at different time steps, defining is the update interval of agent i, and defines the update function as follows:

[0022] (3);

[0023] thus, The expression under the asynchronous update mechanism is:

[0024] (4);

[0025] In the formula, is the control input of agent i at the previous time step (t-1).

[0026] As a preferred embodiment of the present invention, in S13, when When agent i is based on the current state vector (i.e. time t), Calculate the new control input, which is ;when When , agent i maintains the control input of the previous time step .

[0027] As a preferred embodiment of the present invention, in S2, the method of introducing a cancellation model with multiple interferences is:

[0028] S21. If there are p independent interferences in MAS ,but Affected by independent interference, it is expressed as:

[0029] (5);

[0030] In the formula, yes The gain matrix affecting agent i;

[0031] S22, to offset , build a compensation differentiator in the cancellation model , so that:

[0032] (6);

[0033] S23. At this time, and In the asynchronous mechanism, it is expressed as:

[0034] (7);

[0035] S24, will Substituting into formula (1), we get:

[0036] (8);

[0037] S25, after interference compensation, Simplified to:

[0038] (9).

[0039] As a preferred solution of the present invention, in S3, the method of improving the action selection process of the IQL multi-agent system by using a virtual controller is:

[0040] S31. Introduce IQL into MAS for expansion, that is, IQL multi-agent system, and set the state of agent i at time t to , action is , the reward function is , then its standard Q-learning update rule is:

[0041] (10);

[0042] Where α is the learning rate; γ is the discount factor; is the state of agent i at the next time step (t+1), represents the state and action value function of agent i; After iteration ;

[0043] S32. In order to treat each agent as a whole and use the virtual controller to make each agent have mutual correlation when selecting actions, the following action selection strategy is designed: define a correlation matrix ,in represents the correlation between agent i and agent j at time t, The value of is obtained by weighting the output of the virtual controller and is expressed as:

[0044] (11);

[0045] In the formula, and They represent the state estimates of agent i and agent j at time t output by the virtual controller respectively; is the agent (here it can be any agent) under independent interference The state estimation at time ; exp is the exponential function;

[0046] S33, based on , defines the stability parameters of agent i at time t , the update rule is:

[0047] (12);

[0048] In the formula, represents the state and action value function of agent j;

[0049] S34, the improved action selection strategy is expressed as:

[0050] (13).

[0051] As a preferred solution of the present invention, after using a virtual controller to improve the action selection of the IQL multi-agent system, each agent not only considers its own Q value when selecting an action, but also comprehensively considers the Q value of the neighboring agents through the association matrix of the virtual controller, thereby improving the overall coordination and stability, where the Q value is the agent value parameter of IQL, including Q-Table and Q-function.

[0052] As a preferred solution of the present invention, it also includes S5, using Lyapunov candidate function to analyze the stability of MAS and IQL multi-agent systems.

[0053] As a preferred solution of the present invention, for the virtual controller that introduces the offset model, the stability of the MAS is analyzed using the Lyapunov candidate function, and the process is as follows:

[0054] For Lyapunov candidate function , It is expressed as:

[0055] (14);

[0056] In the formula, is a symmetric positive definite matrix; is the time-dependent coupling coefficient, R is a real number set; the superscript T indicates transposition; where, It is expressed as:

[0057] (15);

[0058] In the formula, and is the adjustment parameter; is the control input of agent j at time t;

[0059] The derivative of It is expressed as:

[0060] (16);

[0061] In the formula, yes The first derivative with respect to t;

[0062] Combining equation (9) with equation (12), we get:

[0063] (17);

[0064] In the formula, is a positive definite matrix;

[0065] At this time, for the dynamic asynchronous-based virtual controller, the following conditions are satisfied:

[0066] (18);

[0067] represents partial derivative;

[0068] Therefore, appropriate selection , and , which can make MAS maintain asymptotic stability in the presence of multiple external disturbances and internal coupling disturbances.

[0069] As a preferred solution of the present invention, for an IQL multi-agent system that uses a virtual controller to improve action selection, the stability of the IQL multi-agent system is analyzed using the Lyapunov candidate function, and the process is as follows:

[0070] Lyapunov candidate function for improved IQL multi-agent system It is expressed as:

[0071] (19);

[0072] In the formula, and They represent the target states of agent i and agent j at time t respectively. At this time, the time derivative of the Lyapunov function It is expressed as:

[0073] (20);

[0074] In the formula, , After iteration, and ;

[0075] Combining formula (1), we get:

[0076] (twenty one);

[0077] In the formula, is the interference compensation term; therefore, an appropriate choice , , and (All parameters in the control process can be appropriately selected), and the stability of the IQL multi-agent system is guaranteed.

[0078] The algorithm involved in the present invention can be executed by an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the above algorithm calculation is implemented by executing the software by the processor.

[0079] The beneficial effects of the present invention are:

[0080] The present invention constructs a virtual controller based on dynamic asynchrony. Through the asynchronous update mechanism, different agents are allowed to update their states at different time steps, thereby improving the adaptability of the system (i.e., the IQL multi-agent system) in uncertain environments and effectively solving the problem of collaborative control of the system in complex nonlinear environments.

[0081] The multiple interference cancellation model designed by the present invention can accurately cancel multiple independent interference sources in the system, including the interference of the intelligent agent itself, the interference of other intelligent agents and the interference of the external environment. This design not only enhances the robustness of the system, but also ensures that the system can maintain a stable collaborative control effect under complex interference conditions.

[0082] Based on the traditional IQL algorithm, this paper proposes an improved IQL collaborative control strategy. By introducing the synergy of the association matrix and the virtual controller, each agent is no longer a completely independent individual when selecting an action, but an interconnected system as a whole. This design significantly improves the synergy of the system, avoids the global performance degradation caused by independent decision-making, and ensures the overall optimality of the system.

[0083] The present invention uses Lyapunov function to conduct a rigorous analysis (mathematical proof) of the asymptotic stability of MAS and IQL multi-agent systems, ensuring that the system can stably converge to the target state under multiple disturbances and coupling conditions. In addition, the experimental simulation results further verify the effectiveness of the control strategy under multiple disturbances, showing a significant improvement in response speed and the stability of second-order response fluctuations, all of which prove that the designed controller has good practicality and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 It is a schematic diagram of the process of the present invention;

[0085] Figure 2 The virtual controller is about multiple interferences in the verification process of the present invention. The output response effect diagram is as follows: Figure 2 (a) is the output response effect diagram of the traditional virtual controller; Figure 2 (b), (c), and (d) are output response diagrams of the virtual controller based on dynamic asynchronous in the first, second, and third tests;

[0086] Figure 3 It is a graph showing the output stability of the IQL multi-agent system under triple interference during the verification process of the present invention; Figure 3(a), (b), and (c) are graphs showing the output stability of the IQL multi-agent system in the first, second, and third tests. DETAILED DESCRIPTION

[0087] The embodiments of the present invention are further described below in conjunction with the accompanying drawings:

[0088] like Figure 1 As shown, the IQL collaborative control method of the AI ​​intelligent control system includes the following steps:

[0089] S1. For a multi-agent system MAS consisting of N agents, a virtual controller based on dynamic asynchrony is constructed;

[0090] S2. In the virtual controller, a cancellation model with multiple disturbances is introduced. The cancellation model is used to cancel the multiple external disturbances and internal coupling disturbances existing in the MAS, thereby stabilizing the coordinated control of the MAS.

[0091] S3. For the IQL multi-agent system, the virtual controller is used to improve the action selection process of the IQL multi-agent system, and each agent is equivalent to an interrelated intelligent system rather than an independent individual, thereby improving the control output effect of the IQL multi-agent system under environmental instability;

[0092] S4, based on the improved IQL multi-agent system, realize the coordinated control of N agents;

[0093] S5. Use Lyapunov candidate functions to analyze the stability of MAS and IQL multi-agent systems.

[0094] In S1, the process of building a virtual controller based on dynamic asynchrony is:

[0095] S11. For N agents, the state of agent i at time t is represented by a vector , where i represents the index of the agent, , t represents time, R n represents an n-dimensional real vector space; the control input of agent i at time t is , R m represents an m-dimensional real vector space, then the dynamic behavior of agent i at time t It is represented by the following state space equation:

[0096] (1);

[0097] In the formula, and are the system matrix and input matrix of MAS respectively, The space consisting of real matrices with n rows and m columns; is the nonlinear disturbance term of agent i at time t, describing the uncertainty and nonlinear interference of the environment; this interference comes from other agents on the one hand and from the real-time environment on the other hand;

[0098] S12. Introducing virtual controller , and define it as:

[0099] (2);

[0100] In the formula, is the local feedback gain matrix (which can be designed by the linear quadratic regulator LQP); is the set of agents adjacent to agent i; is the coupling gain matrix between agent i and agent j (which can be designed by a distributed algorithm based on the Laplace matrix L); is the state vector of agent j at time t, and ;

[0101] S13, in order to improve the response speed of MAS in unstable environment, The asynchronous update mechanism is adopted in , that is, different agents can update their states at different time steps, defining is the update interval of agent i, and defines the update function as follows:

[0102] (3);

[0103] thus, The expression under the asynchronous update mechanism is:

[0104] (4);

[0105] In the formula, is the control input of agent i at the previous time step.

[0106] In S13, when When agent i is based on the current state vector Calculate the new control input, which is ;when When , agent i maintains the control input of the previous time step .

[0107] In practical applications It often has multiplicity. In order to enhance the robustness, a cancellation model with multiple disturbances is introduced to optimize the virtual controller.

[0108] In S2, the method of introducing the cancellation model with multiple interferences is:

[0109] S21. If there are p independent interferences in MAS ,but Affected by independent interference, it is expressed as:

[0110] (5);

[0111] In the formula, yes The gain matrix affecting agent i;

[0112] S22, to offset , build a compensation differentiator in the cancellation model , so that:

[0113] (6);

[0114] S23. At this time, and In the asynchronous mechanism, it is expressed as:

[0115] (7);

[0116] S24, will Substituting into formula (1), we get:

[0117] (8);

[0118] S25, after interference compensation, Simplified to:

[0119] (9).

[0120] Compensated differentiators are a state of the art technology that combines the current, previous, and next values ​​of a signal (or more historical data) using a specific algorithm to reduce the effects of noise and provide a more accurate derivative estimate.

[0121] The concept of virtual controller is further extended to improve the action selection process of IQL multi-agent system. By optimizing the stability parameters, each agent no longer operates as a completely independent individual, but through the coordination of the virtual controller, better control output effect can be achieved under environmental instability.

[0122] In S3, the method of using virtual controller to improve the action selection process of IQL multi-agent system is as follows:

[0123] S31. Introduce IQL into MAS for expansion, that is, IQL multi-agent system, and set the state of agent i at time t to , action is , the reward function is , then its standard Q-learning update rule is:

[0124] (10);

[0125] Where α is the learning rate; γ is the discount factor; is the state of agent i at the next time step, Represents the state and action value function of agent i; After iteration ;

[0126] S32. In order to treat each agent as a whole and use the virtual controller to make each agent have mutual correlation when selecting actions, the following action selection strategy is designed: define a correlation matrix ,in represents the correlation between agent i and agent j at time t, The value of is calculated by the output weighting of the virtual controller and is expressed as:

[0127] (11);

[0128] In the formula, and They represent the state estimates of agent i and agent j at time t output by the virtual controller respectively; is the agent in the independent interference The state estimation at time ;

[0129] S33, based on , defines the stability parameters of agent i at time t , the update rule is:

[0130] (12);

[0131] In the formula, represents the state and action value function of agent j;

[0132] S34, the improved action selection strategy is expressed as:

[0133] (13).

[0134] After using the virtual controller to improve the action selection of the IQL multi-agent system, each agent not only considers its own Q value when selecting an action, but also comprehensively considers the Q value of the neighboring agents through the association matrix of the virtual controller, thereby improving the overall coordination and stability. The Q value is the agent value parameter of IQL, including Q-Table and Q-function.

[0135] In S5, for the virtual controller with the offset model introduced, the stability of MAS is analyzed using the Lyapunov candidate function. The process is as follows:

[0136] For Lyapunov candidate function , It is expressed as:

[0137] (14);

[0138] In the formula, is a symmetric positive definite matrix; is the time-dependent coupling coefficient, and R is a set of real numbers; where It is expressed as:

[0139] (15);

[0140] In the formula, and is the adjustment parameter; is the control input of agent j at time t;

[0141] The derivative of It is expressed as:

[0142] (16);

[0143] In the formula, yes The first derivative with respect to t;

[0144] Combining equation (9) with equation (12), we get:

[0145] (17);

[0146] In the formula, is a positive definite matrix;

[0147] At this time, for the dynamic asynchronous-based virtual controller, the following conditions are satisfied:

[0148] (18);

[0149] Therefore, appropriate selection , and , which can make MAS maintain asymptotic stability in the presence of multiple external disturbances and internal coupling disturbances.

[0150] In S5, for the IQL multi-agent system with improved action selection using a virtual controller, the stability of the IQL multi-agent system is analyzed using the Lyapunov candidate function. The process is as follows:

[0151] Lyapunov candidate function for improved IQL multi-agent system It is expressed as:

[0152] (19);

[0153] In the formula, and They represent the target states of agent i and agent j at time t respectively. At this time, the time derivative of the Lyapunov function It is expressed as:

[0154] (20);

[0155] In the formula, , After iteration, and ; Using the dynamic asynchronous update mechanism of the virtual controller, it can be proved that is non-positive, thus ensuring that the IQL multi-agent system is globally asymptotically stable;

[0156] Combining formula (1), we get:

[0157] (twenty one);

[0158] In the formula, is the interference compensation term; therefore, an appropriate choice , , and , the stability of the IQL multi-agent system is guaranteed.

[0159] The verification process is:

[0160] Figure 2 The virtual controller is simulated under multiple disturbances. The output response is Figure 2 From the simulation test results (a) to (d), we can see that the improved virtual controller significantly improves the response speed and stability of the system. Figure 2 In (a), the moment when the response adjustment reaches 0 occurs at 0.86s. Figure 2The moment when the response is adjusted to 0 in (b), (c), and (d) occurs at 0.34s, 0.34s, and 0.38s, respectively. It can be seen that the response efficiency is improved by 0.52s at most, and the fluctuation amplitude of the second-order response is more stable, which proves the effectiveness of the virtual controller of this embodiment in multi-agent collaborative control.

[0161] Figure 2 In the graph, the vertical axis represents the output response of the virtual controller, which is usually the size of the control signal generated by the controller under a given input or feedback. For this control system response curve, the unit of the vertical axis usually depends on the physical quantity of the specific system. Therefore, the specific unit varies according to the actual application scenario. Figure 3 The same applies to the vertical axis.

[0162] Furthermore, three layers of interference are set, namely, interference from the agent itself, interference from other agents, and interference from the external environment. This multiple interference effectively overcomes the defect of independent analysis of agents in IQL and is closer to the real scene. Figure 3 shows this process. Figure 3 In (a), (b) and (c), the control output response to multiple disturbances is shown in Figure 1. Although each figure shows obvious fluctuations at the beginning of the disturbance, especially Figure 3 The fluctuation of (b) is large in the first 2 seconds, but all the graphs can gradually converge to 0, which shows that the system ( Figure 3 The MAS-IQL in the above figure has good self-regulation ability and robustness. No matter how drastic the initial fluctuations are, these response results show the superiority of the proposed control strategy under multiple disturbance conditions, which can effectively stabilize the system and ensure that it eventually tends to a stable state.

Claims

1. IQL collaborative control method of AI intelligent control system, characterized by The following steps are involved: S1. For a multi-agent system MAS consisting of N agents, a virtual controller based on dynamic asynchrony is constructed; S2. In the virtual controller, a cancellation model with multiple disturbances is introduced. The cancellation model is used to cancel the multiple external disturbances and internal coupling disturbances existing in the MAS, thereby stabilizing the coordinated control of the MAS. S3. For the IQL multi-agent system, the virtual controller is used to improve the action selection process of the IQL multi-agent system, and each agent is equivalent to an interrelated intelligent system rather than an independent individual, thereby improving the control output effect of the IQL multi-agent system under environmental instability; S4, based on the improved IQL multi-agent system, realize the coordinated control of N agents; In S1, the process of constructing a virtual controller based on dynamic asynchrony is as follows: S11. For N agents, the state of agent i at time t is represented by a vector , where i represents the index of the agent, , t represents time, R n represents an n-dimensional real vector space; the control input of agent i at time t is , R m represents an m-dimensional real vector space, then the dynamic behavior of agent i at time t It is represented by the following state space equation: （1）； In the formula, and are the system matrix and input matrix of MAS respectively, The space consisting of real matrices with n rows and m columns; is the nonlinear disturbance term of agent i at time t, describing the uncertainty and nonlinear interference of the environment; S12. Introducing virtual controller , and define it as: （2）； In the formula, is the local feedback gain matrix; is the set of agents adjacent to agent i; is the coupling gain matrix between agent i and agent j; is the state vector of agent j at time t, and ; S13, in order to improve the response speed of MAS in unstable environment, The asynchronous update mechanism is adopted in , that is, different agents can update their states at different time steps, defining is the update interval of agent i, and defines the update function as follows: （3）； thus, The expression under the asynchronous update mechanism is: （4）； In the formula, is the control input of agent i at the previous time step; In the above S2, the method of introducing the cancellation model with multiple interferences is: S21. If there are p independent interferences in MAS ,but Affected by independent interference, it is expressed as: （5）； In the formula, yes The gain matrix affecting agent i; S22, to offset , build a compensation differentiator in the cancellation model , so that: （6）； S23. At this time, and In the asynchronous mechanism, it is expressed as: （7）； S24, will Substituting into formula (1), we get: （8）； S25, after interference compensation, Simplified to: （9）； In S3, the method of improving the action selection process of the IQL multi-agent system by using a virtual controller is: S31. Introduce IQL into MAS for expansion, that is, IQL multi-agent system, and set the state of agent i at time t to , Action , the reward function is , then its standard Q-learning update rule is: （10）； Where α is the learning rate; γ is the discount factor; is the state of agent i at the next time step, represents the state and action value function of agent i; After iteration ; S32. In order to treat each agent as a whole and use the virtual controller to make each agent have mutual correlation when selecting actions, the following action selection strategy is designed: define a correlation matrix ,in represents the correlation between agent i and agent j at time t, The value of is calculated by the output weighting of the virtual controller and is expressed as: （11）； In the formula, and They represent the state estimates of agent i and agent j at time t output by the virtual controller respectively; is the agent in the independent interference State estimation at time ; S33, based on , defines the stability parameters of agent i at time t , the update rule is: （12）； In the formula, represents the state and action value function of agent j; S34, the improved action selection strategy is expressed as: （13）。 2. The IQL collaborative control method of the AI ​​intelligent control system according to claim 1 is characterized in that: In the above S13, when When agent i is based on the current state vector Calculate the new control input, which is ;when When , agent i maintains the control input of the previous time step .

3. The IQL collaborative control method of the AI ​​intelligent control system according to claim 1 is characterized in that: After using the virtual controller to improve the action selection of the IQL multi-agent system, each agent not only considers its own Q value when selecting an action, but also comprehensively considers the Q value of the neighboring agents through the association matrix of the virtual controller, thereby improving the overall coordination and stability. The Q value is the agent value parameter of IQL, including Q-Table and Q-function.

4. The IQL collaborative control method of the AI ​​intelligent control system according to claim 1 is characterized in that: It also includes S5, using Lyapunov candidate functions to analyze the stability of MAS and IQL multi-agent systems.

5. The IQL collaborative control method of the AI ​​intelligent control system according to claim 4 is characterized in that: In S5, for the virtual controller with the offset model introduced, the stability of the MAS is analyzed using the Lyapunov candidate function, and the process is as follows: For Lyapunov candidate function , It is expressed as: （14）； In the formula, is a symmetric positive definite matrix; is the time-dependent coupling coefficient, and R is a set of real numbers; where It is expressed as: （15）； In the formula, and is the adjustment parameter; is the control input of agent j at time t; The derivative of It is expressed as: （16）； In the formula, yes The first derivative with respect to t; Combining equation (9) with equation (12), we get: （17）； In the formula, is a positive definite matrix; At this time, for the dynamic asynchronous-based virtual controller, the following conditions are satisfied: （18）； Therefore, appropriate selection , and , which can make MAS maintain asymptotic stability in the presence of multiple external disturbances and internal coupling disturbances.

6. The IQL collaborative control method of the AI ​​intelligent control system according to claim 5 is characterized in that: In S5, for the IQL multi-agent system with improved action selection using a virtual controller, the stability of the IQL multi-agent system is analyzed using the Lyapunov candidate function, and the process is as follows: Lyapunov candidate function for improved IQL multi-agent system It is expressed as: （19）； In the formula, and They represent the target states of agent i and agent j at time t respectively. At this time, the time derivative of the Lyapunov function It is expressed as: （20）； In the formula, , After iteration, and ; Combining formula (1), we get: （21）； In the formula, is the interference compensation term; therefore, an appropriate choice , , and , the stability of the IQL multi-agent system is guaranteed.

Images