Heterogeneous multi-agent system binary output consistency control method

Abstract

The invention discloses a two-way output consistency control method for a heterogeneous multi-agent system, and the method comprises the following steps: 1, constructing state and output equations of a follower agent and a leader agent, calculating an output tracking error ei (k), and constructing an augmented system state space model and a linear control input ui (k); 2, designing a distributed bipartite observer for each follower agent to enable an output tracking error ei (k) to be zero all the time, and replacing a leader agent state of the augmented system state space model with a state of the distributed bipartite observer; 3, defining a Q function, solving the Q function to obtain an optimal control strategy, and introducing a Q-learning algorithm to obtain an optimal control gain only depending on an input state of the augmented system state space model; 4, introducing a fuzzy logic system approximation Q function to enable a matrix phi i to meet a full-rank condition, and updating a fuzzy weight by minimizing a Bellman residual error to obtain an optimal control gain of the input state of the augmented system state space model; and updating the control gain by adopting a gradient descent method, and calculating linear control input ui (k) by utilizing the updated control gain.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of agent system communication, in particular to a heterogeneous multi-agent system bipartite output consensus control method. BACKGROUND

[0002] Multi-agent system is a distributed system composed of multiple agents interacting and communicating in a shared environment. Through coordination and cooperation with each other, multi-agent system can solve complex problems in parallel and significantly improve overall efficiency. In the research of multi-agent system cooperative control, formation, clustering and consensus are the core key problems. Among them, consensus control, as the basis of multi-agent cooperation, aims to make the states of all agents consistent through local interaction. Traditional consensus control methods usually rely on the accurate mathematical model of the controlled object. However, in actual engineering applications, the dynamics of the agent often shows heterogeneous characteristics and is difficult to model accurately, resulting in the failure of many model-based design methods.

[0003] For the distributed optimal output consensus problem of heterogeneous nonlinear multi-agent system under the condition of limited communication, cooperation and confrontation interaction, the existing centralized control based on known model, traditional reinforcement learning relying on global information and model-free adaptive method have the following technical deficiencies: ① Dependence on accurate model or global information: Most optimal control methods require known system dynamics model, while actual systems are often nonlinear and parameters are unknown; Distributed observer often needs global eigenvalue information of Laplace matrix, which is difficult to fully distribute; ② High computational complexity and weak nonlinear processing ability: Traditional Q-learning needs to solve high-dimensional matrix equation iteratively, which has heavy computational burden, and its value function is limited to quadratic form, which is difficult to handle the widely existing nonlinear and heterogeneous dimension problems; ③ Poor stability of strategy update: Control gain update mostly depends on matrix inversion, which has poor numerical stability and is sensitive to initial strategy, which can easily lead to divergence of learning process; ④ Under the condition that the communication topology structure is balanced and the leader information is partially available, there is a lack of complete solution that combines distributed observation, model-free learning and nonlinear approximation, which makes it difficult to guarantee distributed implementation, model-free, efficient computation and nonlinear adaptation.

[0004] Therefore, it is of great theoretical and practical significance to study model-free consensus control algorithm, that is, to achieve cooperative goals without relying on prior knowledge of the system. SUMMARY

[0005] The purpose of the present application is to provide a heterogeneous multi-agent system bipartite output consensus control method to overcome the technical defects in the prior art.

[0006] The technical scheme adopted to achieve the purpose of the present application is:

[0007] A heterogeneous multi-agent system bipartite output consensus control method, comprising the following steps:

[0008] Step 1, a heterogeneous multi-agent system containing N followers and 1 leader is constructed, the state and output equations of the follower agent and the leader agent are constructed respectively, the communication topology of the follower agent in the heterogeneous multi-agent system is described by a directed weighted graph, the cooperation and competition relationship of the follower agent is distinguished, for the follower agent, the characteristic matrix is introduced to divide the follower agent into two subgroups of cooperation and competition, through the diagonal elements d i ={+1,-1} of the characteristic matrix and the output calculation of the follower agent and the leader agent, the output tracking error e i (k) is calculated, the state of the leader and each follower is spliced into the augmented state vector of the follower agent i, the augmented system state space model is constructed, and the linear control input u i (k) is calculated.

[0009] Step 2, for each follower agent in step 1, a distributed bipartite observer is designed to make the output tracking error e i (k) of each follower agent and the output of the leader agent always zero, and the state of the distributed bipartite observer is used to replace the leader agent state of the augmented system state space model in step 1;

[0010] Step 3, based on the discount cost, the Q function U ik (e ik ,ω iik ) is defined, the derivative of the Q function is obtained to obtain the optimal control strategy, the Q-learning algorithm is introduced to solve the Q function to obtain the transformation form i of the Q function matrix Φ , and the optimal control gain

[0011] Step 4, the Q function Q ik (H ik ) is approximated by using a fuzzy logic system, and a double-loop updating mechanism is adopted, in the inner loop, the fuzzy weights are updated by minimizing the Bellman residual, and in the outer loop, the control gain is updated by using the gradient descent method, and the optimal linear control input i (k) is obtained by using the updated control gain to calculate the linear control input u

[0012] In the above technical solution, in step 1, the calculation formula of the linear control input u i (k) is:

[0013] u i (k)=K1i x i (k)+K 2i d i x0(k),

[0014] where K 1i and K 2i are components of the control gain K i for the follower agent i, d i is the diagonal element of the characteristic matrix, x i (k) is the state of the follower agent i at time k, and x0(k) is the target state of the leader at time k;

[0015] The output tracking error e i (k) is represented as:

[0016]

[0017] where d i ={+1,-1} is the diagonal element of the characteristic matrix for the follower agent, y0(k) is the output of the leader agent at time k, and y i (k) is the output of the follower agent i at time k.

[0018] In the above technical solution, in step 1, the augmented system state space model is represented as:

[0019]

[0020]

[0021] where X i (k+1) is the augmented state of the heterogeneous multi-agent system at time k+1, is the augmented state of the heterogeneous multi-agent system at time k, x i (k) is the state of the follower agent i at time k, x0(k) is the state of the leader at time k, and are the augmented system state matrices of the follower agent, is the linear control input of the follower agent i at time k, Y i (k) is the output of the augmented system state space model at time k, A0 is the system matrix of the leader agent, and are the system matrices of the follower agent, is the output matrix of the leader agent, is the output matrix of the follower agent, is the output matrix of the augmented system state space model,

[0022] In the above technical solution, in step 2, the mathematical model of the distributed two-part observer is:

[0023]

[0024] In the formula, M i (k+1) is the parameter of the distributed two-part observer, r i (k) and η i (k) are the state and output of the distributed two-part observer respectively, d i is the diagonal element of the characteristic matrix D, M0 is the augmented matrix of the system matrix and the output matrix of the leader agent, M j (k) is the parameter matrix of the neighbor agent j of the follower agent i, M i (k) is the parameter matrix of the follower agent i, t ij is the element in the i-th row and the j-th column of the normalized adjacency matrix T of the augmented directed signed graph of the heterogeneous multi-agent system, S i (k) is the parameter estimation of the leader agent system matrix, F i (k) is the estimation of the output matrix of the leader agent.

[0025] In the above technical solution, in step 2, the leader state x0(k) of the augmented system state space model after replacement is replaced by the output tracking error e i (k) is expressed as:

[0026]

[0027] In the formula, is the output matrix of the augmented system state space model, X i (k) is the augmented state of the heterogeneous multi-agent system at time k, wherein, r i (k) is the state of the distributed two-part observer;

[0028] The linear control input u i (k) of the heterogeneous multi-agent system is transformed into:

[0029] u i (k)=K 1i x i (k)+K 2i d i x0(k)=K i X i (k)

[0030] In the formula, K iTo control the gain, wherein K i = [K 1i K 2i ].

[0031] In the above technical solution, in step 3, the calculation formula of the Q function Q ik (e ik ,ω ik ) is:

[0032] Q ik (e ik ,ω ik ) = r ik (e ik ,ω ik ) + V ik (e i(k+1) ),

[0033] In the formula, r ik (e ik ,ω iik ) is the utility function of the follower agent i, and V ik (e i(k+1) ) is the cost function of the follower agent i.

[0034] The utility function of the follower agent i The cost function V ik (e ik ) of the follower agent i is r ik (e ik ,ω iik ) + V ik (e i(k+1) ).

[0035] In the above technical solution, in step 3, the calculation formula of the optimal control gain which only depends on the augmented system state space model input state is:

[0036]

[0037] In the formula, -(Φ iωω ) -1 is the negative value of the inverse matrix of the right lower block matrix Φ inside the Q function matrix iωω , and Φ ixω is the right upper block matrix inside the Q function matrix .

[0038] The solution of the transformation matrix of the Q function matrix Φ i is expressed as:

[0039]

[0040] wherein, is the input vector of the fuzzy system.

[0041] In the above technical solution, in the step 4, the update formula of the fuzzy weight is: ik (H ik ) is expressed as:

[0042]

[0043] wherein, ψ i (k) is the FLS weight, is the fuzzy basis function vector, and ε0(H ik ) is the ideal state deviation.

[0044] In the above technical solution, the update formula of the fuzzy weight is:

[0045]

[0046] wherein, α k is the learning rate, ∈ ik is the Bellman residual error.

[0047] In the above technical solution, in the step 4, the update formula of the control gain is:

[0048]

[0049] wherein, β is the learning rate specially designed for the control gain, Q ik is the approximate Q function, wherein Q ik = Q ik (H ik ).

[0050] Compared with the prior art, the beneficial effects of the present application are:

[0051] 1. The present application estimates the leader state online by adopting a distributed dichotomy observer, under the distributed communication framework, first generates a local reference signal online by adopting a distributed dichotomy observer, to process the limited leader information and cooperative-antagonistic interaction, realizes a complete distributed control architecture under the communication limited condition, and effectively solves the problem of dependence on global information in the traditional method.

[0052] 2.The application approximates the action value function by introducing a fuzzy logic system (FLS) to approximate the Q function and combining a double-loop updating mechanism, adopts a dual updating mechanism of Bellman residual error minimization and gradient descent to learn the optimal control gain online, thereby realizing distributed optimal consensus control that is completely model-independent, computationally efficient, nonlinear adaptive, and low in communication demand, effectively improving the convergence, stability, and tracking performance of the system under model uncertainty, nonlinearity, and heterogeneity, and avoiding high-dimensional matrix inversion operations in traditional Q-learning, thereby significantly improving the computational efficiency and numerical stability.

[0053] 3.The application enhances the learning efficiency and convergence performance of the Q-learning algorithm in partially observable environments by adopting an experience replay and adaptive learning rate strategy, greatly reduces the communication and computational resource overhead by retaining only the local control gain and observer during the distributed execution phase, and finally realizes the comprehensive improvement of convergence speed, control accuracy, and system robustness in linear multi-agent systems with unknown models and cooperative-antagonistic interactions, thereby providing effective technical support for practical engineering applications. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 Fig. 1 shows a flowchart of the implementation process of the strategy iteration Q-learning algorithm of the application;

[0055] Figure 2 Fig. 2 shows a flowchart of the implementation process of the improved Q-learning algorithm based on FLS of the application;

[0056] Figure 3 Fig. 3 shows a directed communication topology graph of the leader-follower intelligent agent dynamic system of the application;

[0057] Figure 4 Fig. 4 shows a parameter error trajectory graph of the distributed observer of the application;

[0058] Figure 5 Fig. 5 shows an output trajectory graph of the distributed bisection observer of the application;

[0059] Figure 6 Fig. 6 shows a graph of the components of the intelligent agent control gain under the strategy iteration Q-learning algorithm of the application;

[0060] Figure 7 Fig. 7 shows an output trajectory graph of the intelligent agent state tracking leader of the application;

[0061] Figure 8 Fig. 8 shows a FLS weight parameter trajectory graph of each intelligent agent of the application;

[0062] Figure 9The figure shows the output error trajectory of each agent of the improved Q-learning algorithm of the application based on FLS; DETAILED DESCRIPTION

[0063] The application will be further described below in connection with specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the application and should not be used to limit the application.

[0064] Embodiment 1

[0065] A heterogeneous multi-agent system bipartite output consensus control method, comprising the following steps:

[0066] Step 1, constructing an extended state vector of the heterogeneous system and an output tracking error, and converting the leader-follower bipartite output consensus problem into an output consensus regulation problem of the augmented system, specifically:

[0067] The heterogeneous multi-agent system includes N follower agents and one leader agent, and a directed topological graph is used to describe the communication relationship between agents in the heterogeneous multi-agent system:

[0068] Firstly, the communication topological structure between the N follower agents is described as a weighted directed topological graph where the node set is a non-empty finite set the edge set and the weighted adjacency matrix where (ν j ,ν i )∈ε represents that node i can receive information from node j; the adjacency matrix a ij >0 indicates that the information transmission relationship between node i and node j is a cooperative relationship, a ij <0 indicates a competitive relationship, and a ij =0 means that there is no connection between the two nodes and no information can be received. Set a ii =0; the neighbor set of node ν i is The in-degree matrix of the directed topological graph is a diagonal matrix, defined as where each diagonal element is the weighted in-degree of node i; then the Laplacian matrix of the follower agent directed topological graph is represented as:

[0069]

[0070] The leader agent is marked as ν0, and by introducing the node ν0, the augmented directed signed graph of the heterogeneous multi-agent system is defined as where the node set For leader-follower multi-agent system, further define pinning matrix Λ = diag{a 10 ,…,a N0} refer to Figure 3 , a i0 represent the information transmission from leader agent node v0 to follower agent node v i . When follower agent node v i can receive information from leader agent node v0, a i0 > 0, otherwise a i0 = 0. In the premise of maintaining the generalization ability, assume that the information transmission from leader to follower is unidirectional, i.e. a 0i = 0, then the Laplacian matrix of the augmented directed signed graph of heterogeneous multi-agent system is defined as:

[0071]

[0072] In the formula, -a 1N is the negative value of the first row and the Nth column element of the adjacency matrix , -a N1 is the negative value of the Nth row and the first column element of the adjacency matrix , i is the follower agent index, j is the neighbor index of the follower agent, and N is the number of follower agents.

[0073] Assume that if there is a leader node v0, the leader node v0 is connected to at least all other nodes, then the directed graph contains a spanning tree.

[0074] Divide the node set of the directed topological graph of the follower agent into two parts and and , then the directed graph can be called structurally balanced. When a ij ≥ 0 when or ; a ij < 0 when or . Define a feature matrix D = diag(d1,...,d N ) of a follower agent, which distinguishes the cooperation or competition relationship of the follower agent, wherein the diagonal elements d i = {+1,-1} of the feature matrix represent the camp to which each follower agent belongs.

[0075] Then the follower agent node v i to node v r ​​A directed path is described as a sequence of edges v i ,ν i+1 ,…ν r , where (ν j ,ν i )∈ε, j∈{i,i+1,…r};If there is a node named leader v0, and the directed path from the leader to other nodes is in the directed topological graph, then the graph is defined as containing a spanning tree;

[0076] The dynamics of the follower agent i are as follows:

[0077] x i (k+1)=A i x i (k)+B i u i (k),i=1,2,…,N,(1)

[0078] In the formula, x i (k+1) is the state of agent i at time k+1, is the control of follower agent i; and is the system matrix of the follower agent, it is assumed that A i and B i are unknown, the eigenvalues of A i are located outside or on the edge of the unit circle;

[0079] The output equation of the follower agent i is as follows:

[0080] y i (k)=C i x i (k),(2)

[0081] In the formula, y i (k) is the output of follower agent i at time k; is the output matrix of the follower agent.

[0082] The leader dynamics are as follows:

[0083] x0(k+1)=A0x0(k),(3)

[0084] where A0 is the system input matrix of the leader agent, x0(k+1) is the state of the leader at time k+1, in the augmented directed signed graph of the heterogeneous multi-agent system The leader is at least directly connected to one follower agent.

[0085] The output equation of the leader agent is:

[0086] y0(k) = C0x0(k), (4)

[0087] where, y0(k) is the output of the leader agent at time k; C0is the output matrix of the leader agent.

[0088] Designing the linear control input u i (k) for the heterogeneous multi-agent system is:

[0089] u i (k) = K 1i x i (k) + K 2i d i x0(k), (5)

[0090] where K 1i and K 2i are components of the control gain of the follower agent i, d i is the diagonal element of the characteristic matrix, x i (k) is the state of the follower agent i at time k, and x0(k) is the state of the leader at time k.

[0091] Since the nodes of the follower agents are divided into two stable subgroups according to the cooperation and competition relationship and the consistency control goal of the present application is not only to make all the follower outputs consistent with the leader, but also to make the follower output track the output of the leader or the opposite value of the output of the leader, i.e., the goal is to achieve the bipartite consistency, which is specifically:

[0092] According to the diagonal element d i ={+1,-1} of the characteristic matrix of the follower agent, the output y0(k) of the leader agent at time k, and the output y i (k) of the follower agent i at time k, the output tracking error of the leader agent and the follower agent is calculated i.e., when the leader and the follower belong to the same camp, when the leader and the follower belong to different camps,

[0093] In order to make the expression more concise and the formula more suitable for the structure of the heterogeneous bipartite system, let where d i= {+1, -1} are diagonal elements of the feature matrix D, representing the camp to which each agent belongs. The heterogeneous multi-agent system is converted into a heterogeneous multi-agent augmented system, and the augmented state of the heterogeneous multi-agent system is denoted as

[0094] The goal of the consensus control is to obtain the optimal control gain K of each follower agent i using the information from the follower agent i itself and its neighbor follower agents * and the optimal linear control output to make the outputs of all follower agents eventually and the output of the leader reach a two-thirds consensus, i.e.

[0095] According to the state of the augmented system of the heterogeneous multi-agent, the output tracking error of the follower agents and the leader agent is converted into

[0096]

[0097] wherein, X i (k) is the augmented state of the heterogeneous multi-agent system at time k.

[0098] The linear control input u of the heterogeneous multi-agent system i (k) becomes:

[0099] u i (k) = K 1i x i (k) + K 2i d i x0(k) = K i X i (k), (7)

[0100] wherein, K i is a control gain, wherein K i = [K 1i K 2i ], (k) is the augmented state of the heterogeneous multi-agent system at time k.

[0101] Referring to Figure 3 , the state of the augmented system space model of the heterogeneous multi-agent is represented as:

[0102]

[0103] wherein, (k) is the augmented state of the heterogeneous multi-agent system at time k, and ​is the augmented system state space model matrix of the follower agent, is the linear control input of the follower agent i at time k. i (k) is the output of the augmented system state space model at time k, is the augmented system state space model matrix of the follower agent,

[0104] Step 2, in the actual environment, not all follower agents can receive the leader agent signal, for the communication constraint problem that the leader state information in the multi-agent system cannot be directly obtained by all followers, a distributed dichotomy observer is designed for each follower agent i in step 1, and the mathematical model of the distributed dichotomy observer is:

[0105]

[0106] In the formula, M i (k+1) is the parameter of the distributed dichotomy observer, r i (k) and η i (k) are the state and output of the distributed dichotomy observer, d i is the diagonal element of the characteristic matrix D. It is defined that t ij is the normalized adjacency matrix T of the augmented directed signed graph of the heterogeneous multi-agent system The element in the i-th row and the j-th column of the normalized adjacency matrix T of the augmented directed signed graph of the heterogeneous multi-agent system, Under the assumption, the output of the distributed dichotomy observer converges to the output of the leader or its opposite at an exponential rate, that is, or When the observer converges, r i (k) can replace the leader state x0(k) in the state of the augmented system state space model of the heterogeneous multi-agent, and the augmented state of the heterogeneous multi-agent system at time k is

[0107] The distributed dichotomy observer has three important characteristics: first, it does not depend on the global characteristic information of the Laplacian matrix, and realizes a completely distributed architecture; second, it explicitly handles cooperative and antagonistic interactions through a sign transformation factor and a sign function, and is suitable for balanced directed graphs; third, it can maintain estimation performance even if the communication between the leader agent and some follower agents is interrupted, and has strong robustness. The distributed dichotomy observer can generate local reference signals r i (k) and η i (k) for each follower agent without relying on global topology information. Referring to Figure 4 , Figure 5 are the parameter error trajectory graph and the output trajectory graph of the distributed dichotomy observer of the application, respectively;

[0108] Step 3, after the observer is designed, the output tracking error of the follower agent and the leader agent Define the test control input ω i (k) = mK i X i (k) and the utility function U of the follower agent of the tracking error ik (e ik , ω iik ), construct the Q function based on the discounted cost, and give the optimal control strategy expression derived based on the Q function Bellman equation; design a data-driven algorithm based on Q-learning, collect system running data X ik , ω ik , X i(k+1) , ω i(k+1) , and use the Q function Bellman equation to solve the Q function matrix Φ i by least squares iteration Get the optimal control gain Substitute the optimal control gain into the linear control output in step 1 to get the optimal control strategy This method eliminates the limitation that the optimal control rate must rely on the system dynamic model in the past.

[0109] The utility function U of the follower agent i ik (e ik , ω iik ) is:

[0110]

[0111] In the formula, e ik is the abbreviation of e i (k), the output tracking error of the follower agent and the leader agent, ω iik is the abbreviation of ω i (k), the test control input ω i (k) = mK i X i (k) of the follower agent i.

[0112] Define the cost function V of the follower agent i ik (e ik ) as:

[0113]

[0114] In the formula, U im (e im , ω im) is the utility function of the follower agent i at time m, e im is the output tracking error of the follower agent and the leader agent at time m, ω im is the test input of agent i at time m, i is the follower agent, and m is the time.

[0115] Define a unique positive definite matrix P i , then the cost function V ik (e ik ) of the follower agent i is expressed as a quadratic form as follows:

[0116]

[0117] According to the Bellman optimality principle, the cost function V ik (e ik ) can be expressed as:

[0118] V ik (e ik ) = r ik (e ik , ω iik ) + V ik (e i(k+1) ), (14)

[0119] In addition, define the Q function Q ik (e ik , ω ik ) as follows:

[0120] Q ik (e ik , ω ik ) = r ik (e ik , ω ik ) + V ik (e i(k+1) ), (15)

[0121] Therefore, the Q function can be written as:

[0122]

[0123] Because , the Q function is expressed as:

[0124]

[0125] By taking the derivative of the Q function with respect to the test control input ω ik , the optimal control policy expression is obtained:

[0126]

[0127] Because the optimal control gain is solved using the Bellman optimality principle. It will depend on the system matrix (i.e. Therefore, in solving model-free problems, the Q-learning algorithm is introduced to construct a model solely from the system's input-output states and control data.

[0128] Continue with the test input ω i (k)=mK i X i Substituting (k) into formula (17), we get:

[0129]

[0130] make in, Define Φ i Let Φ be the Q-function matrix. i It is expressed as follows:

[0131]

[0132] Therefore, the Q function can be rewritten as:

[0133]

[0134] Because Q ik (e ik ,ω ik ) = V ik (e ik Therefore, we can deduce that:

[0135]

[0136] Therefore, the Q-function matrix can be obtained by the following formula:

[0137]

[0138] definition Therefore, formula (23) can be rewritten as:

[0139]

[0140] In the formula, φ ij For matrix Φ i The component in the i-th row and j-th column; Among them, h ik(j) For H ik The j-th component.

[0141] Solving formula (24) using the least squares method, we get: in, for The only solution.

[0142] Based on the matrix Φ of the Q function i To find the optimal gain, then

[0143] Matrix Φ i In solving It is necessary to ensure that matrix Φ i Only by achieving full rank can the state-action space be fully explored, that is... To satisfy this condition, the test control input ω ik Add exploration signal δ ik That is, continuous excitation, so that the test control input ω ik The state-action space can be fully explored.

[0144] Reference Figure 1 Here is a specific example of step 3:

[0145] Step 1), Initialization: Initialize the iteration value j = 0, Q function matrix Φ 0 Arbitrary coupling gain matrix K 0 Set the test control input, δ ik To explore the signal, the maximum number of training loop steps is set to 100;

[0146] Step 2): During the j-th iteration, collect R sets of data (X) at time k. ik ,ω ik ,X i(k+1) ,ω i(k+1) ), where ω i(k+1) Through ω i(k+1) =mK j-1 X ik We obtain, and the dataset satisfies

[0147] Step 3), Policy estimation: Solve for the Q-function matrix Φ using the Bellman equation. j :

[0148]

[0149] Step 4), Strategy Update: via the Q-function matrix Φ j K is obtained from the internal matrix block. j and update ω ik =mK j X ik +δ ik ,in

[0150] Step 4, refer to Figure 2, in order to overcome the inherent defects of traditional Q-learning method in dealing with high-dimensional state-action space, such as high computational complexity and poor numerical stability, a fuzzy logic system is introduced to construct an efficient function approximator, and an improved algorithm is proposed: at the algorithm implementation level, the data set X ik ,ω ik ,X i(k+1) is collected through system interaction, and the Q function is approximated by a fuzzy logic system (FLS) where, is the input vector of the fuzzy system, is the carefully designed fuzzy basis function vector, ψ i is the adjustable weight vector. Compared with the traditional Q-learning method which needs to construct and update a high-dimensional Q matrix, this fuzzy approximation mechanism reduces the computational complexity from O((n i +n0+m i ) 3 ) to O(L 2 ), where L is the number of fuzzy rules, and L<<n i +n0+m i , which significantly improves the computational efficiency.

[0151] The FLS based on IF-THEN rules can be expressed as:

[0152] Rule R l :

[0153] Then:Q ik is P l

[0154] where, is the FLS input, j is the jth dimension of H ik vector, and Q ik is the FLS output. is the fuzzy set of the jth dimension in the lth rule, P l is the output fuzzy set is the membership function of , and is the membership function designed for each dimension of H ik , defined as:

[0155]

[0156] where, is the membership function center, is the membership function width. In addition, we define as the normalized fuzzy basis function vector:

[0157]

[0158] where L denotes the total number of rules. Thus the approximation function Q ik (H ik ) is:

[0159]

[0160] The approximation Q function where

[0161] The learning process is implemented by using a double-loop updating mechanism: in the inner loop (Critic updating), the fuzzy weights are updated by using the Bellman residual minimization mechanism, and the updating formula is where a k is the learning rate. This process realizes the model-free learning of the optimal action value function. Specifically,

[0162] Let Then the gradient calculation formula is The updating formula is where a k is the learning rate, and e ik is the Bellman residual. This process realizes the model-free learning of the optimal action value function.

[0163] where the Bellman residual is calculated by

[0164]

[0165] The error of the updated modulus weight is referenced Figure 8 .

[0166] In the outer loop (Actor updating), the control gain matrix is updated by using the gradient descent method based on the learned Q function. The gradient descent method The control gain updating formula is: where b is the learning rate designed specifically for the control gain. The optimal linear control output is calculated according to the updated control gain Compared with the traditional matrix inversion method, this gradient descent mechanism has better numerical stability and effectively avoids the algorithm failure problem caused by the singularity of the Q block matrix.

[0167] The main process is as follows:

[0168] Step 1), initialization: randomly initialize the FLS weights i (0), the control gain K i (0), and design the fuzzy rules and membership functions of the FLS, and initialize the experience replay pool.

[0169] Step 2), exploration and data collection: generate test input ω ik = mK 0 X ik + δ ik and observe the next state X i(k+1) and reward r ik Store the data into the experience replay pool.

[0170] Step 3), FLS weight update (minimize Bellman residual): randomly collect a batch of data, calculate Bellman residual ik :

[0171] To minimize the residual, gradient descent is used to minimize the following formula: and Continue to update the FLS weight: where α k is the learning rate, denotes the size of the experience stack, which must satisfy Let According to the literature, when , since the recorded data has linear independence, at this time rank(N) = l

[0172] Step 4), control gain update: calculate the gradient of the Q function to the gain:

[0173]

[0174] Update the control gain:

[0175]

[0176] Referring to Figure 6 , the figure of each component of the control gain of the agent in the strategy iteration Q-learning algorithm of the application is shown, and finally the updated control gain is substituted into u i (k) in step 1 to act on the heterogeneous multi-agent system, referring to Figure 7 , Figure 9 , the output tracking error of each follower agent and leader agent calculated based on the FLS improved Q-learning algorithm is 0.

[0177] In order to further improve the stability and convergence performance of the learning process, the application also introduces a variety of optimization techniques. In the data collection stage, the experience replay mechanism is used to construct a data buffer to store historical interaction experience, and by random sampling, the correlation between data is broken, satisfying the continuous incentive condition. In the parameter update stage, an adaptive learning rate adjustment strategy is designed, which dynamically adjusts kThe values of and β can accelerate the learning speed while ensuring convergence. In addition, through the optimization design of the number of fuzzy rules and the careful selection of the Q function parameters, the best balance between approximation accuracy and computational efficiency is achieved.

[0178] In the system deployment and execution phase, each follower only needs to keep the local optimal gain and the distributed observer. The agent uses the local interaction information X i (k), according to the optimal linear control output The distributed control input is generated without the coordination of the central controller and the exchange of control gain or value function estimate with other agents, which has small communication overhead and good privacy protection. When the system dynamics change or the communication topology switches, the distributed dichotomy observer and the fuzzy Q-learning algorithm can be updated online, automatically adapt to environmental changes, and do not need to redesign the controller, which shows good adaptive ability and robustness.

[0179] The above only describes the preferred embodiments of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A binary output consistency control method for a heterogeneous multi-agent system, characterized in that, The method comprises the following steps: Step 1, construct a heterogeneous multi-agent system containing N followers and 1 leader, construct the state and output equations of the follower agent and the leader agent respectively, and use a directed weighted graph to describe the communication topology of the follower agent in the heterogeneous multi-agent system, distinguish the cooperation and competition relationship of the follower agent, for the follower agent, introduce the characteristic matrix to divide the follower agent into two subgroups of cooperation and competition, through the diagonal elements d i ={+1,-1} of the characteristic matrix and the output calculation of the follower agent and the leader agent, the output tracking error e i (k) is calculated, the state of the leader and each follower is spliced into the augmented state vector of the follower agent i, the augmented system state space model is constructed, and the linear control input u i (k) is calculated; Step 2, for each follower agent in Step 1, design a distributed dichotomic observer to make each follower agent's output track the error e of the leader agent's output i (k) is always zero, and the state of the distributed dichotomic observer is used to replace the leader agent's state of the augmented system state space model in Step 1; Step 3, define Q function U based on discounted cost ik (e ik ,ω iik ), derive the optimal control strategy by deriving Q function, introduce Q-learning algorithm to solve Q function to obtain Q function matrix Φ i transformation form value, obtain the optimal control gain only dependent on the input of the augmented system state space model Step 4, approximating the Q function Q with a fuzzy logic system ik (H ik ), and using a double-loop updating mechanism, in the inner loop, the fuzzy weights are updated by minimizing the Bellman residual error, and in the outer loop, the control gain is updated using the gradient descent method, and the updated control gain is used to calculate the linear control input u i (k) in Step 1 to obtain the optimal linear control input 2. The heterogeneous multi-agent system bisected output consensus control method of claim 1, wherein, In step 1, the linear control input u i (k) is calculated according to the formula u i (k) = K 1i x i (k) + K 2i d i x0(k), where K 1i and K 2i are components of the control gain K i for the follower agent i, d i is the diagonal element of the feature matrix, x i (k) is the state of the follower agent i at time k, and x0(k) is the target state of the leader at time k. The output tracking error e i (k) is represented as: where d i = {+1,-1} is the diagonal element of the characteristic matrix of the follower agent, y0(k) is the output of the leader agent at time k, and y i (k) is the output of the follower agent i at time k.

3. The heterogeneous multi-agent system bisected output consensus control method of claim 1, wherein, In the step 1, the augmented system state space model is expressed as: where X i (k+1) is the augmented state of the heterogeneous multi-agent system at time k + 1, is the augmented state of the heterogeneous multi-agent system at time k, x i (k) is the state of the follower agent i at time k, x0(k) is the state of the leader at time k, and is the augmented system state matrix of the follower agent, is the linear control input of the follower agent i at time k, Y i (k) is the output of the augmented system state space model at time k, A0is the system matrix of the leader agent, and is the system matrix of the follower agent, is the output matrix of the leader agent, is the output matrix of the follower agent, is the output matrix of the augmented system state space model, 4. The heterogeneous multi-agent system bisected output consensus control method of claim 1, wherein, In the step 2, the mathematical model of the distributed dichotomous observer is: where M i (k+1) is the parameter of the distributed binary observer, r i (k) and η i (k) are the state and output of the distributed binary observer, respectively, d i is the diagonal element of the feature matrix D, M0is the augmented matrix of the system matrix and output matrix of the leader agent, M j (k) is the parameter matrix of the neighbor agent j of the follower agent i, M i (k) is the parameter matrix of the follower agent i, t ij is the element of the i-th row and j-th column of the normalized adjacency matrix T of the augmented directed signed graph of the heterogeneous multi-agent system, S i (k) is the parameter estimation of the leader agent system matrix, F i (k) is the estimation of the leader agent output matrix.

5. The heterogeneous multi-agent system bisected output consensus control method of claim 1, wherein, The leader state xo(k) of the augmented system state space model after replacement in step 2 is outputted after the tracking error e i (k) is represented as: wherein is the output matrix of the augmented system state space model X i (k) is the augmented state of the heterogeneous multi-agent system at time k, where r i (k) is the state of the distributed dichotomic observer Linear control input u for a heterogeneous multi-agent system i (k) transforms into: u i (k) = K 1i x i (k) + K 2i d i (k) = K i X i (k) where K i is a control gain, and K i = [K 1i K 2i ].

6. The heterogeneous multi-agent system bisected output consensus control method of claim 1, wherein, In step 3, the calculation formula of the Q function Q ik (e ik ,ω ik ) is as follows: Q ik (e ik ,ω ik )=r ik (e ik ,ω ik )+V ik (e i(k+1) ), where r ik (e ik ,ω iik ) is the utility function of the follower agent i, V ik (e i(k+1) ) is the cost function of the follower agent i; The follower agent i's utility function The follower agent i's cost function V ik (e ik ) = r ik (e ik , ω iik ) + V ik (e i(k+1) ).

7. The heterogeneous multi-agent system bisected output consensus control method of claim 1, wherein, In the step 3, the calculation formula of the optimal control gain only depending on the augmented system state space model input state is as follows: The calculation formula is as follows. wherein - (Φ iωω ) -1 is the Q-function matrix internal right lower block matrix Φ iωω negative of the inverse matrix, Φ ixω is the Q-function matrix internal right upper block matrix; The Q function matrix Φ i The transformation matrix The solution is represented as: wherein is the input vector to the fuzzy system.

8. The heterogeneous multi-agent system bisected output consensus control method of claim 1, wherein, In step 4, the approximate Q-function Q ik (H ik ) is expressed as: where ψ i (k) is the FLS weight, is the fuzzy basis function vector, ε0(H ik ) is the ideal state deviation.

9. The heterogeneous multi-agent system bisected output consensus control method of claim 8, wherein, The updating formula of the fuzzy weight is: In the formula, α k is a learning rate, ∈ ik is a Bellman residual.

10. The heterogeneous multi-agent system bisected output consensus control method of claim 1, wherein, In the step 4, the updating formula of the control gain is: where β is a learning rate designed specifically for the control gain, Q ik is an approximation of Q function, where Q ik = Q ik (H ik ).

Images