Double-time-scale multi-agent system control method and system based on reinforcement learning

Abstract

The invention relates to the technical field of multi-agent systems, and provides a double-time-scale multi-agent system control method and system based on reinforcement learning, and the method comprises the steps: building a tracking error between agents based on a communication topological graph, carrying out the derivation to obtain the dynamic information of the tracking error, combining hostile attacks with control input, and carrying out the control of the hostile attacks. An integral sliding mode controller is adopted to counteract a coupling term to obtain a tracking error dynamic model, an H-infinity control method is adopted to convert the tracking error dynamic model into a zero-sum game problem between agents, then a Hamiltonian function related to performance indexes is defined, partial derivative of the Hamiltonian function is solved to obtain a tracking strategy, the tracking strategy is substituted into a Lyapunov equation, and the tracking error dynamic model is obtained. And obtaining a game algebraic Riccati equation of the tracking error, and after splitting, calculating to obtain the optimal feedback gain of the multi-agent system by using an integral reinforcement learning algorithm through a strategy iteration mode. The malicious attack is eliminated while the optimal consistency of the multi-agent system is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-agent systems, and in particular to a dual-time-scale multi-agent system control method and system based on reinforcement learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, the consistency problem of multi-agent systems has attracted extensive attention from researchers due to its wide application in sensor networks, autonomous aircraft formation control, and multi-robot collaboration. Consistency and stability are important prerequisites and foundations for collaborative control research. Consistency refers to the state in which all agents interact with each other and provide feedback based on the designed distributed controller, and finally reach the same state. Distributed control methods integrate the local information of each agent and its neighbors to enable all agents to operate in coordination, thereby providing the advantages of stronger performance and flexibility. In practical engineering fields, such as robot control systems, power systems, circuit systems, etc., slow and fast dynamic coupling problems are often encountered. Such systems are also called singular perturbation systems. Since the phenomenon caused by singular perturbation systems is also called dual time scale phenomenon, combining dual time scale with multi-agent systems has become the exploration direction of many researchers. Secondly, it is inevitable that the agents will encounter attacks during communication during the control process, which will make the control signal unable to control the system state in time, resulting in reduced controller efficiency and even system crash; in addition, from the perspective of optimal control, it is hoped that the best control effect can be obtained with lower control energy, thereby reducing system loss. However, in practical applications, the information of the system is not completely unknown, so the reinforcement learning method with unknown system information is used. Therefore, it is of great research value and significance to study the sliding mode consistency control problem of dual-time-scale multi-agent system based on reinforcement learning.

[0004] In the design of control schemes for multi-agent systems, several core issues need to be considered. The first is how to deal with malicious attacks, and the second is how to design the optimal controller to achieve the effect of asymptotically zeroing the errors between multi-agents and reaching a consensus. For the first problem, the publication number CN112327632A, the patent "A tracking control method for a multi-agent system with false data injection attacks" respectively adopted a network predictive control scheme based on a Kalman filter to deal with the malicious attack problem in the multi-agent system, but did not consider the problem of optimal control. For the second problem, the publication number CN7608190A, the patent "An optimal consistency protocol design method for a discrete multi-agent system" adopts a discrete time linear augmented system based on predictive control theory, and obtains the feedback gain matrix of the augmented system according to the quadratic optimal control theory, thereby solving the optimal consistency problem. However, it does not consider the consistency control problem of a dual-time scale multi-agent system with an unknown system model. Therefore, it is crucial to study the sliding mode consistency control problem of a dual-time scale multi-agent system based on reinforcement learning.

[0005] At present, there are many studies on multi-agent systems, and these studies have improved the performance of multi-agent systems to a certain extent and achieved good results. However, there are no related research topics that consider malicious attack processing and multi-agent consistency issues at the same time. Summary of the invention

[0006] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a dual-time-scale multi-agent system control method and system based on reinforcement learning. It takes the multi-agent system as the object, eliminates attacks by using a sliding mode control method, takes approximate strategy iterative learning as the basic method, and takes performance optimization and multi-agent consistency as the fundamental goals, thereby achieving the elimination of malicious attacks while ensuring the optimal consistency of the multi-agent system.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A first aspect of the present invention provides a dual-time-scale multi-agent system control method based on reinforcement learning, which comprises:

[0009] Obtain multi-agent system parameters, establish inter-agent communication topology and multi-agent system model;

[0010] Based on the communication topology, the tracking error between agents is established, the tracking error is derived, and the dynamic information of the tracking error is obtained by combining the multi-agent system model. Then, the malicious attack with control input is combined to obtain the coupling term. The integral sliding mode controller is used to offset the coupling term, and the tracking error dynamics model of the dual-time scale multi-agent system is obtained.

[0011] For the tracking error dynamics model, H ∞ The control method converts the optimal consensus problem of the multi-agent system into a zero-sum game problem between agents, defines the Hamiltonian function related to the performance indicator, takes the partial derivative of the Hamiltonian function, derives the tracking strategy, and introduces the tracking strategy into the Lyapunov equation to obtain the game algebraic Riccati equation of the tracking error. After splitting the game algebraic Riccati equation, the integral reinforcement learning algorithm is used to calculate the optimal feedback gain of the multi-agent system through strategy iteration.

[0012] Furthermore, the tracking error is:

[0013]

[0014] Among them, i (t) is the tracking error of agent i, N is the number of neighbors of agent i, and x i (t) is the position of agent i, x j (t) represents the position of agent j, the neighbor of agent i.

[0015] Furthermore, the dynamic information of the tracking error is:

[0016]

[0017] Among them, i (t) is the tracking error of agent i, N is the number of neighbors of agent i, and A ε With B εi is the multi-agent system parameter, u i (t) represents the control amount of agent i, u j (t) represents the control amount of neighbor agent j of agent i, Ψ i (x i (t),t) represents a malicious attack, x i (t) is the position of agent i, x j (t) represents the position of neighbor agent j of agent i, and ω(t) is the coupling term.

[0018] Furthermore, the integral sliding mode surface of the integral sliding mode controller is:

[0019]

[0020] Among them, Γ is a positive definite matrix, ζ i (t) is the tracking error of agent i, A ε With B εi is the control parameter of the multi-agent system, u i0 (t) is the feedback gain control strategy, u j(t) represents the control amount of neighbor agent j of agent i.

[0021] Furthermore, the tracking strategy includes the optimal control strategy of the agent and the worst control strategy of its neighbors.

[0022] A second aspect of the present invention provides a dual-time-scale multi-agent system control system based on reinforcement learning, comprising:

[0023] A model building module, which is configured to: obtain multi-agent system parameters, establish a communication topology diagram between agents and a multi-agent system model;

[0024] The attack elimination module is configured as follows: based on the communication topology diagram, the tracking error between the agents is established, the tracking error is derived, and after the dynamic information of the tracking error is obtained by combining the multi-agent system model, the malicious attack with the control input is merged to obtain the coupling term, and the coupling term is offset by the integral sliding mode controller to obtain the dual-time scale multi-agent system tracking error dynamics model;

[0025] The optimization control module is configured as follows: for the tracking error dynamics model, H ∞ The control method converts the optimal consensus problem of the multi-agent system into a zero-sum game problem between agents, defines the Hamiltonian function related to the performance indicator, takes the partial derivative of the Hamiltonian function, derives the tracking strategy, and introduces the tracking strategy into the Lyapunov equation to obtain the game algebraic Riccati equation of the tracking error. After splitting the game algebraic Riccati equation, the integral reinforcement learning algorithm is used to calculate the optimal feedback gain of the multi-agent system through strategy iteration.

[0026] Furthermore, the dynamic information of the tracking error is:

[0027]

[0028] Among them, i (t) is the tracking error of agent i, N is the number of neighbors of agent i, and A ε With B εi is the multi-agent system parameter, u i (t) represents the control amount of agent i, u j (t) represents the control amount of neighbor agent j of agent i, Ψ i (x i (t),t) represents a malicious attack, x i (t) is the position of agent i, x j (t) represents the position of neighbor agent j of agent i, and ω(t) is the coupling term.

[0029] Furthermore, the integral sliding mode surface of the integral sliding mode controller is:

[0030]

[0031] Among them, Γ is a positive definite matrix, ζ i (t) is the tracking error of agent i, A ε With B εi is the control parameter of the multi-agent system, u i0 (t) is the feedback gain control strategy, u j (t) represents the control amount of neighbor agent j of agent i.

[0032] The third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the dual-time-scale multi-agent system control method based on reinforcement learning as described above.

[0033] The fourth aspect of the present invention provides a computer device, comprising a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, wherein when the processor executes the program, the steps in the dual-time-scale multi-agent system control method based on reinforcement learning as described above are implemented.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention takes the multi-agent system as the object, eliminates attacks by using the sliding mode control method, takes approximate strategy iterative learning as the basic method, and takes performance optimization and multi-agent consistency as the fundamental goals, thereby achieving the elimination of malicious attacks while ensuring the optimal consistency of the multi-agent system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0037] Figure 1 is a flow chart of a dual-time-scale multi-agent system control method based on reinforcement learning according to Embodiment 1 of the present invention;

[0038] Figure 2 A communication topology diagram between intelligent agents according to the first embodiment of the present invention;

[0039] Figure 3 The first embodiment of the present invention uses the reinforcement learning method to obtain With the optimal value The difference graph;

[0040] Figure 4 The first embodiment of the present invention uses the reinforcement learning method to obtain With the optimal value The difference between the graph and the

[0041] Figure 5 is a state diagram of an intelligent agent according to the first embodiment of the present invention;

[0042] Figure 6 A state diagram of an intelligent agent in a three-dimensional plane according to the first embodiment of the present invention;

[0043] Figure 7 It is a structural diagram of a computer device according to a fourth embodiment of the present invention. DETAILED DESCRIPTION

[0044] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0045] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0046] Embodiment 1

[0047] This embodiment provides a dual-time-scale multi-agent system control method based on reinforcement learning.

[0048] The dual-time-scale multi-agent system control method based on reinforcement learning provided in this embodiment carries out sliding mode control, model-free consistency optimal control and other problems for the dual-time-scale multi-agent system. It takes the model-free dual-time-scale multi-agent system as the object, eliminates attacks by the sliding mode control method, takes approximate strategy iterative learning as the basic method, takes distributed control as the basic means, takes performance optimization and consistency as the fundamental goals, and establishes a theoretical and methodological system of dual-time-scale multi-agent distributed optimization control of reinforcement learning.

[0049] The dual-time-scale multi-agent system control method based on reinforcement learning provided in this embodiment solves the optimal consistency problem of the dual-time-scale multi-agent system based on the reinforcement learning method; in order to resist attacks, an integral sliding mode control method is adopted to eliminate the impact of attacks on the multi-agent system; malicious attacks can be eliminated while ensuring the optimal consistency of the multi-agent system.

[0050] The dual-time-scale multi-agent system control method based on reinforcement learning provided in this embodiment is as follows: Figure 1 As shown, the following steps are included:

[0051] Step 1: For the first-order multi-agent system with malicious attacks, graph theory is used to establish a communication topology graph between the first-order multi-agent systems.

[0052] The communication topology graph is an undirected graph, that is, the agent can obtain the information of the adjacent agents and send its own information to the adjacent agents.

[0053] Among them, the dual-time-scale multi-agent system model with malicious attacks is expressed as:

[0054]

[0055] in, and Respectively represent the state quantity of the fast subsystem and the state quantity of the slow subsystem; represents the state of the i-th agent in the multi-agent system; u i (t) represents the control quantity of the i-th agent in the multi-agent system; x i (t) is a 2-row by 1-column matrix, x i1 (t) and x i2 (t) represents x i The first and second rows of (t) are written separately to express the dual time scale phenomenon later; ε represents the singular perturbation parameter and 0<ε<<1.

[0056]

[0057] Among them, A ε With B εi is a constant matrix (multi-agent system parameters).

[0058] Step 2: According to the communication topology established in step 1, establish the tracking error between agent i and adjacent agent j in the multi-agent system, that is, the tracking error is equal to the position x of agent i i (t) Subtract the position x of the neighboring agent j j (t); At the same time, the tracking error is derived to obtain the dynamic information of the tracking error, and then the control input u i (t) and u j (t) is used to merge the coupling items of malicious attacks. The specific steps are as follows:

[0059] Step 201, define the tracking error between agents as follows:

[0060]

[0061] Where N represents the number of neighbors of agent i.

[0062] Step 202: Deriving the tracking error, the dynamic information of the tracking error is obtained as follows:

[0063]

[0064] in, is represented as an in-degree matrix and if Then a ij > 0, otherwise a ij =0; Represents a network communication topology graph consisting of n nodes, where

[0065] It is the set of all nodes and points in the communication topology graph. represents the set of edges, and (v i ,v j )(i≠j) indicates that agent i can communicate with its neighbor j; define the Laplacian matrix The element is l ij , where l ii =d i , l ij =-a ij ,Right now ζ i (t) represents the error between the ith agent and its neighboring agent j.

[0066] Step 203: Consider malicious attack as Ψ i (x i (t),t), the control input is rewritten as follows:

[0067] u ia (t) = u i (t)+Ψ(x i (t),t) (4)

[0068] Therefore, the dynamic information of agent i is rewritten as follows:

[0069] x i (t) = A ε x i (t)+B εi u ia (t) (5)

[0070] Therefore, the dynamic information of the tracking error is rewritten as follows:

[0071]

[0072] Among them, the control strategy is u i (t) = u i0 (t)+uis (t),u i0 (t) is the feedback gain control strategy, u is (t) is the feedforward gain control strategy, and ω(t) is the coupling term of the attack.

[0073] Step 3: Use an integral sliding mode controller for the control input u i (t) and u j The coupling term of the malicious attack of (t) is offset, which proves that the sliding mode control method can eliminate the impact of malicious attacks. Finally, a new dynamic model equation of the tracking error of the dual-time scale multi-agent system can be obtained.

[0074] Specifically, the following integral sliding surface is defined:

[0075]

[0076] where Γ is a positive definite matrix.

[0077] The integral sliding mode control strategy is as follows:

[0078] u is (t) = -α(ΓB εi ) -1 Sgn(s i (ζ i (t),t)) (8)

[0079] where α is a positive definite matrix, Sgn(·) is a sign function; u is (t) represents the integral sliding mode control strategy, which is used to continuously iterate to eliminate the impact of malicious attacks on the system.

[0080] In this embodiment, the Lyapunov stability theory is used to verify the validity of the integral sliding surface (Formula (7)), which proves that the sliding mode controller can offset the attack, so that there is no attack influence in the dynamic information of the error system.

[0081] In step 3, the malicious attack is eliminated by the integral sliding mode strategy, so the tracking error dynamics model formula is a nominal system model formula, namely:

[0082] Step 4: Based on the dual-time-scale multi-agent system tracking error dynamics model obtained in step 3, use H ∞ The control method converts the optimal consensus problem of the multi-agent system into a zero-sum game problem between agents. Then, for the zero-sum game problem between agents, a performance index function and Hamiltonian function of the tracking error are proposed, and the partial derivative of the Hamiltonian function is obtained to obtain the optimal control strategy of the error system. Worst-case control strategy Then the tracking strategy (optimal control strategy Worst-case control strategy ) is substituted into the Lyapunov equation, and the game algebraic Riccati equation of the error system is obtained.

[0083] In this embodiment, the error system performance indicator function is defined as follows:

[0084]

[0085] Among them, Q i >0、R ii >0 and R ij >0; λ is a positive scalar, indicating the change from the disturbance input To the defined performance output variables The attenuation of the matrix R ii With R ij is the weight matrix of the control quantity and is symmetrical; R ii represents the agent i itself, R ij The matrix Q represents the j neighboring agent i i is the weight matrix of the performance index function for the state quantity, which is a diagonal matrix.

[0086] Define the error system cost function as Among them, P εi is a positive definite real symmetric matrix and is the solution to the error system cost function.

[0087] Therefore, the Hamiltonian equation relating the performance index is as follows:

[0088]

[0089] Therefore, the optimal control strategy of agent i and the worst control strategies of its neighbors are as follows:

[0090]

[0091] Substituting the optimal control strategy (11) and the worst control strategy (12) into the Hamiltonian equation (9), we finally get the game algebraic Riccati equation as follows:

[0092]

[0093] In order to avoid calculation errors, P εi This simplifies to the following: in,

[0094] At this time, formula (13) can be rewritten as follows:

[0095]

[0096] Step 5: For the game algebraic Riccati equation obtained in step 4, a full-order method is proposed to process the singular perturbation parameters to avoid the calculation error problem caused by the dual time scale; the theorem is given and proved to ensure that the multi-agent system can reach a consensus; then the game algebraic Riccati equation is split, and an integral reinforcement learning algorithm is used to calculate the optimal feedback gain of the multi-agent system through strategy iteration. and

[0097] In order to solve Solve it using the following Lyapunov equation:

[0098]

[0099] Where k represents the number of iterations, and As shown below:

[0100]

[0101] From formula (14) to formula (16), we can know that The derivative of is given by:

[0102]

[0103] Integrating both sides of formula (18), the result is as follows:

[0104]

[0105] Define the following matrix:

[0106]

[0107] Where, t≤t0<...<t s <t+ΔT.

[0108] The parameters in the above matrix are as follows:

[0109]

[0110]

[0111] Among them, o=1,2,3,...,s.

[0112] In order to simplify the calculation, formula (18) is rewritten into the form of Kronecker product as follows:

[0113]

[0114] in, and According to formula (19), the following equation can be obtained:

[0115]

[0116] The parameters in the above formula are as follows:

[0117]

[0118] in, and if is of full rank, then the unique solution of equation (19) is given by:

[0119]

[0120] In summary, the design steps for the optimal controller for the consistency of a linear dual-time-scale multi-agent system, that is, the strategy iteration method, are as follows:

[0121] (1) Let i = 1 and select the initial stable feedback gain and Used as input in the learning process, where d is the detection noise; calculate and collect data;

[0122] (2) According to formula (24), find and

[0123] (3) Let i = i + 1, and repeat step (2) until and i>1, where parameter σ>0 is a predefined cutoff threshold.

[0124] Step 6: Substitute the proposed control algorithm into the simulation model for testing to verify that the control algorithm can be used for optimal consistency control of dual-time-scale multi-agent systems and has good results.

[0125] The method of the present invention is simulated and analyzed below. By writing a Matlab program to solve the linear equation to obtain the controller gain, it is verified that the control algorithm can be used in a dual-time-scale multi-agent system with attacks and has good effects.

[0126] In order to prove that the reinforcement learning method designed in this example for the consistency of a dual-time-scale multi-agent system can achieve consensus control behavior against a multi-agent system with malicious attacks, a simulation experiment is given here.

[0127] In the simulation, the expression of the multi-agent system with dual time scales is given as follows:

[0128]

[0129] Among them, the above matrix parameters are as follows:

[0130] Set the initial value of the agent as follows: x1 = [1,1] T , x2=[5,5] T , x3=[10,10] T , x4=[-5,-5] T , x5=[-10,-10] T .

[0131] Among them, Q1=3I2, Q2=4I2, Q3=2I2, Q4=3I2 and Q5=5I2.

[0132] Make all agents use a common value R ij =2I and λ=3.5.

[0133] In order to prevent the agent from diverging, the initial stable gain is given as follows: The attack parameter is Ψ(x i (t),t)=cos(x i1 (t)), the sliding mode parameters are Γ=0.1, α=[2,2] T .

[0134] Figure 2 It is the communication topology diagram of the intelligent agent. In this experiment, the communication between intelligent agents is carried out according to this diagram. Figure 3 and Figure 4 It represents P iterated by reinforcement learning algorithm i ε(k) , Compared with the optimal solution P calculated by the traditional Lyapunov method i ε , It can be seen that after the fifth iteration, their error can be less than 1×10 -5 , indicating that the P i ε , Optimal solution. Figure 5 Indicates the state of each agent. It can be seen that an attack is added to agent 1, and then the sliding mode controller is introduced at 0.5s. After the impact of the attack is eliminated, the optimal value after learning is used at 2s, and then the agents reach a consensus. For the convenience of observation, Figure 6A 3D graph is used to represent the trajectories of the agents, from which it can be seen that the agents finally reach a consensus. Based on the above simulation results, the following conclusions can be drawn: the sliding mode consensus control scheme for the dual-time-scale multi-agent system based on reinforcement learning proposed in the present invention is effective, and has the advantages of fast response, high efficiency and strong robustness.

[0135] Embodiment 2

[0136] This embodiment provides a dual-time-scale multi-agent system control system based on reinforcement learning, which specifically includes:

[0137] A model building module, which is configured to: obtain multi-agent system parameters, establish a communication topology diagram between agents and a multi-agent system model;

[0138] The attack elimination module is configured as follows: based on the communication topology diagram, the tracking error between the agents is established, the tracking error is derived, and after the dynamic information of the tracking error is obtained by combining the multi-agent system model, the malicious attack with the control input is merged to obtain the coupling term, and the coupling term is offset by the integral sliding mode controller to obtain the dual-time scale multi-agent system tracking error dynamics model;

[0139] The optimization control module is configured as follows: for the tracking error dynamics model, H ∞ The control method converts the optimal consensus problem of the multi-agent system into a zero-sum game problem between agents, defines the Hamiltonian function related to the performance indicator, takes the partial derivative of the Hamiltonian function, derives the tracking strategy, and introduces the tracking strategy into the Lyapunov equation to obtain the game algebraic Riccati equation of the tracking error. After splitting the game algebraic Riccati equation, the integral reinforcement learning algorithm is used to calculate the optimal feedback gain of the multi-agent system through strategy iteration.

[0140] It should be noted here that each module in this embodiment corresponds to each step in Example 1 one by one, and the specific implementation process is the same, which will not be repeated here.

[0141] Embodiment 3

[0142] This embodiment provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the dual-time-scale multi-agent system control method based on reinforcement learning as described in the above-mentioned embodiment 1.

[0143] Embodiment 4

[0144] This embodiment provides a computer device, such as Figure 7As shown, it includes a display device, an input device, a computer-readable storage medium (volatile memory and non-volatile storage medium), a processor, a communication interface (i.e., a network interface), and a computer program stored on the computer-readable storage medium and executable on the processor, wherein the processor, the communication interface, and the computer-readable storage medium can be connected via a bus or other means. The communication interface is used to receive and send data, and when the processor executes the program, the steps in the dual-time-scale multi-agent system control method based on reinforcement learning as described in the first embodiment above are implemented.

[0145] Among them, any reference to memory, storage, database or other media provided by the present application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0146] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0149] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A dual-time-scale multi-agent system control method based on reinforcement learning, characterized in that: include: Obtain multi-agent system parameters, establish inter-agent communication topology and multi-agent system model; Based on the communication topology, the tracking error between agents is established, the tracking error is derived, and the dynamic information of the tracking error is obtained by combining the multi-agent system model. Then, the malicious attack with control input is combined to obtain the coupling term. The integral sliding mode controller is used to offset the coupling term, and the tracking error dynamics model of the dual-time scale multi-agent system is obtained. For the tracking error dynamics model, H ∞ The control method converts the optimal consensus problem of the multi-agent system into a zero-sum game problem between agents, defines the Hamiltonian function related to the performance indicator, takes the partial derivative of the Hamiltonian function, derives the tracking strategy, and introduces the tracking strategy into the Lyapunov equation to obtain the game algebraic Riccati equation of the tracking error. After splitting the game algebraic Riccati equation, the integral reinforcement learning algorithm is used to calculate the optimal feedback gain of the multi-agent system through strategy iteration.

2. The dual-time-scale multi-agent system control method based on reinforcement learning as claimed in claim 1, characterized in that: The tracking error is: Among them, i (t) is the tracking error of agent i, N is the number of neighbors of agent i, and x i (t) is the position of agent i, x j (t) represents the position of agent j, the neighbor of agent i.

3. The dual-time-scale multi-agent system control method based on reinforcement learning as claimed in claim 1, characterized in that: The dynamic information of the tracking error is: Among them, i (t) is the tracking error of agent i, N is the number of neighbors of agent i, and A ε With B εi is the multi-agent system parameter, u i (t) represents the control amount of agent i, u j (t) represents the control amount of neighbor agent j of agent i, Ψ i (x i (t),t) represents a malicious attack, x i (t) is the position of agent i, x j (t) represents the position of neighbor agent j of agent i, and ω(t) is the coupling term.

4. The dual-time-scale multi-agent system control method based on reinforcement learning as claimed in claim 1, characterized in that: The integral sliding mode surface of the integral sliding mode controller is: Among them, Γ is a positive definite matrix, ζ i (t) is the tracking error of agent i, A ε With B εi is the control parameter of the multi-agent system, u i0 (t) is the feedback gain control strategy, u j (t) represents the control amount of neighbor agent j of agent i.

5. The dual-time-scale multi-agent system control method based on reinforcement learning as claimed in claim 1, characterized in that: The tracking strategy includes the optimal control strategy of the agent and the worst control strategy of its neighbors.

6. A dual-time-scale multi-agent system control system based on reinforcement learning, characterized in that: include: A model building module, which is configured to: obtain multi-agent system parameters, establish a communication topology diagram between agents and a multi-agent system model; The attack elimination module is configured as follows: based on the communication topology diagram, the tracking error between the agents is established, the tracking error is derived, and after the dynamic information of the tracking error is obtained by combining the multi-agent system model, the malicious attack with the control input is merged to obtain the coupling term, and the coupling term is offset by the integral sliding mode controller to obtain the dual-time scale multi-agent system tracking error dynamics model; The optimization control module is configured as follows: for the tracking error dynamics model, H ∞ The control method converts the optimal consensus problem of the multi-agent system into a zero-sum game problem between agents, defines the Hamiltonian function related to the performance indicator, takes the partial derivative of the Hamiltonian function, derives the tracking strategy, and introduces the tracking strategy into the Lyapunov equation to obtain the game algebraic Riccati equation of the tracking error. After splitting the game algebraic Riccati equation, the integral reinforcement learning algorithm is used to calculate the optimal feedback gain of the multi-agent system through strategy iteration.

7. The dual-time-scale multi-agent system control system based on reinforcement learning as claimed in claim 6, characterized in that: The dynamic information of the tracking error is: Among them, i (t) is the tracking error of agent i, N is the number of neighbors of agent i, and A ε With B εi is the multi-agent system parameter, u i (t) represents the control amount of agent i, u j (t) represents the control amount of neighbor agent j of agent i, Ψ i (x i (t),t) represents a malicious attack, x i (t) is the position of agent i, x j (t) represents the position of neighbor agent j of agent i, and ω(t) is the coupling term.

8. The dual-time-scale multi-agent system control system based on reinforcement learning as claimed in claim 6, characterized in that: The integral sliding mode surface of the integral sliding mode controller is: Among them, Γ is a positive definite matrix, ζ i (t) is the tracking error of agent i, A ε With B εi is the control parameter of the multi-agent system, u i0 (t) is the feedback gain control strategy, u j (t) represents the control amount of neighbor agent j of agent i.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the dual-time-scale multi-agent system control method based on reinforcement learning as described in any one of claims 1-5 are implemented.

10. A computer device comprising a computer-readable storage medium, a processor, and a computer program stored in the computer-readable storage medium and executable on the processor, characterized in that: When the processor executes the program, the steps in the dual-time-scale multi-agent system control method based on reinforcement learning as described in any one of claims 1-5 are implemented.

Images