Multi-alliance game agent control method and device, equipment and medium

Abstract

The invention discloses an agent control method and device for a multi-alliance game, equipment and a medium, and relates to the technical field of agent control, and the method comprises the steps: dividing agents in an unmanned cluster system into a plurality of alliances; the method comprises the following steps: establishing a directed unbalanced communication topological graph of the unmanned cluster system based on actual communication conditions among intelligent agents in the unmanned cluster system; establishing a multi-alliance game agent decision state dynamical model; determining a target alliance according to the multi-alliance game agent decision state dynamical model; determining a Laplacian matrix of the target alliance and an estimation vector of a target agent in the target alliance; determining an average gradient estimation item of the target agent; determining a Nash equilibrium search term of the target agent according to the average gradient estimation term of the target agent; and controlling the agents in the unmanned cluster system according to the Nash equilibrium search terms of all the target agents in all the target alliances. According to the invention, rapid control of the unmanned cluster is realized.

Description

Technical Field

[0001] The present application relates to the field of intelligent agent control technology, and in particular to an intelligent agent control method, device, equipment and medium for multi-coalition games. Background Art

[0002] With the development and application of communication networking technology and unmanned system technology, the research on multi-agent systems has received widespread attention and application. In recent years, the research on multi-coalition game algorithms has become a focus issue. The purpose of the game is to obtain greater benefits. In the game, an important concept and strategy set is the Nash equilibrium. The Nash equilibrium describes an optimal strategy set for different agents in the game. In this state, any agent unilaterally changes its decision and cannot obtain greater benefits. Therefore, each agent changes its own decision state in the game to make its alliance obtain greater benefits.

[0003] An alliance is composed of agents with common interests; multiple agents with different interests form multiple alliances. Unlike the basic inter-individual game problem, in the multi-alliance game problem, cooperation and game relations coexist. In the multi-alliance game, each agent forms a game alliance due to its own interest relationship. Within the alliance, each agent cooperates with each other, optimizes the alliance cost function, and competes with other alliances. The alliance can be regarded as a virtual individual participating in the game of other alliances, but the actual decision makers are still the agents participating in the game.

[0004] In practice, the speed of algorithm convergence is an important evaluation indicator of algorithm performance. The preset time algorithm can preset an expected algorithm convergence time in advance. By designing the time-varying control gain, the control algorithm completes convergence within the preset time. In addition, due to differences in signal transmission power and communication channel establishment, directed unbalanced communication topologies are more common in practice. However, most related methods are aimed at simpler communication topologies.

[0005] Therefore, eliminating the interference caused by the directed unbalanced communication topology of the unmanned cluster and improving the efficiency of the intelligent agent converging to the Nash equilibrium solution will help the intelligent agent to iteratively evolve to the optimal decision-making state faster in the game, thereby achieving rapid control of the unmanned cluster. Summary of the invention

[0006] The purpose of this application is to provide an intelligent agent control method, device, equipment and medium for multi-coalition games, which can eliminate the interference caused by the directed unbalanced communication topology of the unmanned cluster system and improve the efficiency of the intelligent agent converging to the Nash equilibrium solution, and provide a fast solution method for the unmanned cluster control problem from the perspective of game theory, thereby realizing the rapid control of the unmanned cluster.

[0007] To achieve the above objectives, this application provides the following solutions:

[0008] In a first aspect, the present application provides an agent control method for a multi-coalition game, comprising:

[0009] The intelligent agents in the unmanned swarm system are divided into a plurality of alliances; each of the alliances comprises: a plurality of intelligent agents;

[0010] Based on the actual communication situation between each intelligent agent in the unmanned swarm system, a directed unbalanced communication topology diagram of the unmanned swarm system is established;

[0011] Establish a decision-making state dynamics model of multi-coalition game agents based on all agents in the alliance;

[0012] Determine the target alliance according to the decision state dynamics model of the multi-alliance game agent; the target alliance is an alliance with agents that meet the trigger conditions; the agent that meets the trigger conditions in the target alliance is the target agent;

[0013] According to the directed unbalanced communication topology graph, determining the Laplacian matrix of the target alliance and the estimated vector of the target agent in the target alliance; the estimated vector converges to the left zero eigenvector of the Laplacian matrix of the target alliance to which the target agent belongs at a first preset time;

[0014] Determining an average gradient estimation term of the target agent according to the estimation vector of the target agent; wherein the average gradient estimation term converges to an average estimation value of the overall cost function of the target alliance to which the target agent belongs within a second preset time;

[0015] Determining a Nash equilibrium search item for the target intelligent agent according to the average gradient estimation item of the target intelligent agent; the Nash equilibrium search item converges to a Nash equilibrium solution of the target intelligent agent within the second preset time; the Nash equilibrium solution is a decision state vector that minimizes the overall cost function of the target alliance to which the target intelligent agent belongs;

[0016] The agents in the unmanned swarm system are controlled according to the Nash equilibrium search items of all target agents in all target coalitions.

[0017] Optionally, the expression of the decision state dynamics model of the multi-coalition game agent is:

[0018]

[0019] y ij =C ij x ij ;

[0020] Among them, yij represents the decision state vector of the jth agent in the i-th alliance, y ij ∈R q , R q represents a q-dimensional real vector, u ij represents the control input of the jth agent in the ith coalition, Indicates p ij dimensional real vector, x ij represents the internal state of the jth agent in the ith alliance, Indicates n ij dimensional real vector, represents the update rate of the internal state of the jth agent in the ith alliance, A ij represents the system dynamics matrix of the jth agent in the ith alliance, B ij represents the system input matrix of the jth agent in the ith alliance, C ij represents the decision matrix of the jth agent in the ith coalition.

[0021] Optionally, based on the actual communication situation between the agents in the unmanned cluster system, a directed unbalanced communication topology diagram of the unmanned cluster system is established, specifically including:

[0022] Based on the actual communication between the agents in the unmanned cluster system, the communication topology within the i-th alliance is established. in, represents the set of all agents participating in the i-th alliance, m i represents the number of agents in the ith alliance, represents the first agent in the ith alliance, Indicates the mth in the ith alliance i Agents; E i represents the communication edge set of agents in the ith alliance, represents a communication edge from the jth agent in the ith alliance to the kth agent in the ith alliance, represents the jth agent in the i-th alliance, represents the kth agent in the i-th alliance;

[0023] The weighted adjacency matrix of the ith alliance is established according to the communication topology within the ith alliance. The weighted adjacency matrix is ​​a square matrix. When , the value of the element in the kth row and jth column of the weight adjacency matrix of the i-th alliance is otherwise

[0024] According to the weighted adjacency matrix of the ith alliance, determine the in-degree of the jth agent in the ith alliance and the out-degree of the jth agent in the ith alliance; the in-degree of the jth agent in the ith alliance Represents the value of the element in the jth row and kth column of the weighted adjacency matrix of the i-th alliance, and the out-degree of the j-th agent in the i-th alliance

[0025] The directed unbalanced communication topology graph of the unmanned cluster is constructed by the in-degree of all agents in the i-th alliance and the out-degree of all agents.

[0026] Optionally, determining the Laplacian matrix of the target alliance and the estimated vector of the target agent in the target alliance according to the directed unbalanced communication topology graph specifically includes:

[0027] Determine the Laplace matrix of the target alliance; the expression of the Laplace matrix is:

[0028]

[0029] Among them, L i is the Laplacian matrix of the i-th target alliance, D i is a diagonal matrix constructed by the out-degrees of all agents in the i-th target alliance, is the weighted adjacency matrix of the ith coalition;

[0030] A first preset time gain operator is established; the expression of the first preset time gain operator is:

[0031]

[0032] Wherein, h>1, h is the gain rate control parameter of the first preset time gain operator, T is the second preset time, φ(t;T) is the first preset time gain operator with the preset time T, and t is time;

[0033] The estimated vector of the target agent in the target alliance is determined according to the first preset time gain operator and the directed unbalanced communication topology graph; the expression of the estimated vector is:

[0034]

[0035] Among them, z ij is the estimated vector of the jth target agent in the i-th target coalition, is the update rate of the estimation vector of the jth target agent in the i-th target alliance, φ(t;∈T) is the first preset time gain operator with preset time ∈T, ∈T is the first preset time, 0<∈<1.

[0036] Optionally, determining the average gradient estimation term of the target agent according to the estimation vector of the target agent specifically includes:

[0037] A second preset time gain operator is established; the expression of the second preset time gain operator is:

[0038]

[0039] in, is the second preset time gain operator, φ(t; T) is the first preset time gain operator with the preset time being T;

[0040] Initialize auxiliary variables of the jth target agent in the i-th target coalition The kth element of and the average gradient estimate g ij The kth element of is 0, i∈{1,2,…,N}, N is the number of alliances participating in the game, j,k∈{1,2…,m i};

[0041] According to the estimation vector of the target agent, the average gradient estimation term of the target agent is determined; the expression of the average gradient estimation term of the target agent is:

[0042]

[0043] in, f ij () is the cost function of the jth target agent in the i-th target alliance, y represents the set of decision state vectors of all target agents in all target alliances, η represents the set of Nash equilibrium search items of all target agents in all target alliances, η i represents the set of Nash equilibrium search items of all target agents in the i-th target alliance, N is the number of target alliances participating in the game, represents the mth target alliance in the i-th target alliance i The Nash equilibrium search items of the target agent, Indicates m i q-dimensional real vector, for The update rate, z ijj is the estimated vector z of the jth target agent in the i-th target alliance ij The jth element of is the second preset time gain operator, g ijk is the average gradient estimate g of the jth target agent in the i-th target alliance ij The kth element of .

[0044] Optionally, determining the Nash equilibrium search term of the target agent according to the average gradient estimation term of the target agent specifically includes:

[0045] Initializing the Nash equilibrium search item of the target agent to a zero vector;

[0046] According to the average gradient estimation item of the target agent, the Nash equilibrium search item of the target agent is determined; the expression of the Nash equilibrium search item of the target agent is:

[0047]

[0048] Among them, η ij is the Nash equilibrium search item of the jth target agent in the i-th target alliance, is η ij The update rate of g is, κ is the Nash equilibrium search rate control parameter, κ>0, g ijj is the average gradient estimate g of the jth target agent in the i-th target alliance ij The j-th element of .

[0049] Optionally, controlling the agents in the unmanned cluster system according to the Nash equilibrium search items of all target agents in all target alliances specifically includes:

[0050] The first gain matrix and the second gain matrix are constructed according to the decision state dynamics model of the multi-coalition game agent; the calculation formulas of the first gain matrix and the second gain matrix are as follows:

[0051]

[0052] Among them, K ij1 is the first gain matrix of the jth target agent in the i-th target alliance, K ij2 is the second gain matrix of the jth target agent in the i-th target alliance, I q is a q×q-dimensional identity matrix with all diagonal elements being 1 and all other elements being 0;

[0053] According to the Nash equilibrium search items of all target agents in all target alliances, the first gain matrix and the second gain matrix, the control input of all target agents in all target alliances is obtained; the formula of the control input of the target agent is:

[0054]

[0055] Among them, u' ij is the control input of the jth target agent in the i-th target coalition;

[0056] The agents in the unmanned swarm system are controlled according to the control inputs of all target agents in all target alliances.

[0057] In a second aspect, the present application provides an intelligent agent control device for a multi-coalition game, comprising:

[0058] The alliance division module is used to divide the intelligent agents in the unmanned cluster system into multiple alliances; each of the alliances includes: multiple intelligent agents;

[0059] A directed unbalanced communication topology diagram establishment module is used to establish a directed unbalanced communication topology diagram of the unmanned cluster system based on the actual communication conditions between various intelligent agents in the unmanned cluster system;

[0060] A model building module is used to build a decision-making state dynamics model of multi-alliance game agents based on the agents in all alliances;

[0061] A target alliance determination module is used to determine a target alliance according to the decision state dynamics model of the multi-alliance game agent; the target alliance is an alliance with agents that meet the trigger conditions; the agent in the target alliance that meets the trigger conditions is the target agent;

[0062] An estimation vector determination module is used to determine the Laplace matrix of the target alliance and the estimation vector of the target agent in the target alliance according to the directed unbalanced communication topology graph; the estimation vector converges to the left zero eigenvector of the Laplace matrix of the target alliance to which the target agent belongs at a first preset time;

[0063] an average gradient estimation item determination module, configured to determine an average gradient estimation item of the target agent according to the estimation vector of the target agent; the average gradient estimation item converges to an average estimation value of the overall cost function of the target alliance to which the target agent belongs within a second preset time;

[0064] A Nash equilibrium search item determination module, used to determine the Nash equilibrium search item of the target intelligent agent according to the average gradient estimation item of the target intelligent agent; the Nash equilibrium search item converges to the Nash equilibrium solution of the target intelligent agent within the second preset time; the Nash equilibrium solution is a decision state vector that minimizes the overall cost function of the target alliance to which the target intelligent agent belongs;

[0065] A control module is used to control the agents in the unmanned cluster system according to the Nash equilibrium search items of all target agents in all target alliances.

[0066] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the agent control method for multi-alliance game described in any one of the above.

[0067] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned methods for controlling an intelligent agent in a multi-alliance game.

[0068] According to the specific embodiments provided in this application, this application has the following technical effects:

[0069] The present application provides an agent control method, device, equipment and medium for a multi-alliance game, which determines an estimation vector through a directed unbalanced communication topology diagram of an unmanned cluster. Through the determination of the estimation vector, the problem of weight mismatch caused by directly adopting communication between agents under the directed unbalanced communication topology diagram in the subsequent average gradient estimation item is solved, thereby failing to obtain the average gradient estimation item; then, based on the average gradient estimation item, the limitation of the agent being able to obtain only its own cost function and needing to optimize the entire alliance cost function is solved, and a Nash equilibrium search item is designed in combination with the estimation vector; based on the Nash equilibrium search item, the target agent is driven to converge to a Nash equilibrium solution within a second preset time, thereby improving the efficiency of the agent converging to the Nash equilibrium solution, and helping the agent to iteratively evolve to an optimal decision-making state faster in the game, thereby achieving rapid control of the unmanned cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 This is a flow chart of an intelligent agent control method for a multi-coalition game in one embodiment of the present application;

[0072] Figure 2 A schematic diagram of a communication topology relationship provided in an embodiment of the present application;

[0073] Figure 3 A schematic diagram of the decision trajectory of agents in a multi-coalition game;

[0074] Figure 4 It is a schematic diagram of the estimation error curve for the left zero eigenvector;

[0075] Figure 5 A schematic diagram of functional modules of an intelligent agent control device for multi-alliance gaming provided by an embodiment of the present application;

[0076] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0079] Regarding the multi-coalition game algorithm, Ye et al. proposed a multi-coalition game Nash equilibrium search algorithm based on dynamic average consistency estimation and singular perturbation principle to estimate the average consistency of gradients. Liu et al. introduced the penalty function method and studied the multi-coalition game algorithm for second-order agents under constraints. Deng et al. studied the multi-coalition game Nash equilibrium search algorithm for second-order agents in a more general communication situation of a balanced directed graph. However, it is worth noting that these algorithms can only obtain results of asymptotic or exponential convergence, and are all for relatively simple communication topologies. In practice, the speed of algorithm convergence is an important evaluation indicator of the performance of the algorithm, and the preset time algorithm can preset an expected algorithm convergence time in advance. By designing a time-varying control gain, the control algorithm completes convergence at a preset time. In addition, due to differences in signal transmission power and communication channel establishment, unbalanced directed communication topology graphs are more common in practice. Therefore, this application proposes an agent control method for multi-coalition games to solve the impact brought by directed unbalanced communication topology graphs and realize rapid control of unmanned clusters.

[0080] In an exemplary embodiment, Figure 1 As shown, a method for controlling an intelligent agent in a multi-coalition game is provided, comprising:

[0081] S1: The intelligent agents in the unmanned swarm system are divided into multiple alliances; each alliance includes: multiple intelligent agents. Specifically, the alliances are divided according to the cooperation and competition relationship reflected by the cost function.

[0082] S2: Based on the actual communication between the agents in the unmanned swarm system, a directed unbalanced communication topology diagram of the unmanned swarm system is established. Specifically, it includes:

[0083] Based on the actual communication between the agents in the unmanned cluster system, the communication topology within the i-th alliance is established. in, represents the set of all agents participating in the i-th alliance, m i represents the number of agents in the ith alliance, m i ≥2, represents the first agent in the ith alliance, Indicates the mth in the ith alliance i Agents; E i represents the communication edge set of agents in the ith alliance, represents a communication edge from the jth agent in the ith alliance to the kth agent in the ith alliance, represents the jth agent in the i-th alliance, represents the kth agent in the ith alliance. If two agents communicate, there is an edge between the nodes corresponding to the two agents. If there is a pair of corresponding edges, it satisfies Then the graph is a directed graph.

[0084] The weighted adjacency matrix of the ith alliance is established according to the communication topology within the ith alliance. The weighted adjacency matrix is ​​a square matrix. When , the value of the element in the kth row and jth column of the weight adjacency matrix of the i-th alliance is otherwise

[0085] According to the weighted adjacency matrix of the ith alliance, determine the in-degree of the jth agent in the ith alliance and the out-degree of the jth agent in the ith alliance; the in-degree of the jth agent in the ith alliance Represents the value of the element in the jth row and kth column of the weighted adjacency matrix of the i-th alliance, and the out-degree of the j-th agent in the i-th alliance If there is a node whose out-degree is not equal to its in-degree, then the graph is a directed unbalanced graph.

[0086] The directed unbalanced communication topology graph of the unmanned cluster is constructed by the in-degree of all agents in the i-th alliance and the out-degree of all agents.

[0087] Consider the communication topology between alliances It describes the communication relationship between alliances, which is similar to the implementation scheme recorded in the above method and will not be repeated here. The communication topology of the entire multi-alliance game is That is, communication within the alliance + communication between alliances.

[0088] S3: Establish a decision-making state dynamics model of multi-alliance game agents based on the agents in all alliances.

[0089] The dynamics of each agent in the system is driven by a model of the following form. The expression of the decision state dynamics model of multi-coalition game agents is:

[0090]

[0091] Among them, y ij represents the decision state vector of the jth agent in the i-th alliance, y ij ∈R q , R q represents a q-dimensional real vector, u ij represents the control input of the jth agent in the ith coalition, Indicates p ij dimensional real vector, x ij represents the internal state of the jth agent in the ith alliance, Indicates n ij dimensional real vector, represents the update rate of the internal state of the jth agent in the ith alliance, A ij represents the system dynamics matrix of the jth agent in the ith alliance, B ij represents the system input matrix of the jth agent in the ith alliance, C ij represents the decision matrix of the jth agent in the ith coalition.

[0092] S4: Determine the target alliance based on the decision-making state dynamics model of multi-alliance game agents; the target alliance is an alliance with agents that meet the trigger conditions; the agent that meets the trigger conditions in the target alliance is the target agent. The trigger condition is C ij B ij The line is full of rank.

[0093] S5: Determine the Laplacian matrix of the target alliance and the estimated vector of the target agent in the target alliance according to the directed unbalanced communication topology graph; the estimated vector converges to the left zero eigenvector of the Laplacian matrix of the target alliance to which the target agent belongs at a first preset time.

[0094] In this step, for the established communication topology The Laplace matrix L i , establish the Laplace matrix L iThe estimated vector of the left zero eigenvector of is used to eliminate the influence of weight mismatch caused by direct communication between agents under the unbalanced directed communication topology when estimating the average gradient term in the subsequent process, thus failing to obtain the average gradient estimate. According to the directed unbalanced communication topology, the Laplacian matrix of the target alliance and the estimated vector of the target agent in the target alliance are determined, including:

[0095] Determine the Laplacian matrix of the target alliance; the expression of the Laplacian matrix is:

[0096]

[0097] Among them, L i is the Laplacian matrix of the i-th target alliance, D i is a diagonal matrix constructed by the out-degrees of all agents in the i-th target alliance, is the weighted adjacency matrix of the ith coalition.

[0098] A first preset time gain operator is established; the expression of the first preset time gain operator is:

[0099]

[0100] in, It is the left derivative of μ(t) and is the intermediate auxiliary variable for constructing φ(t; T). h>1, h is the gain rate control parameter of the first preset time gain operator, h is a constant. The larger the value, the greater the gain. φ(t; T) is the first preset time gain operator with a preset time of T. T is the second preset time and can be set to any value. t is time (i.e., the current moment when the algorithm is running, for example, if it has run for 3s, t is 3). The first preset time gain operator is a function related to time t, and time is the independent variable.

[0101] Initialize the parameter value z in the estimation vector of the jth target agent in the i-th target alliance ijj (0)=1,z ijk (0) = 0, k ≠ j, where z ijj Represents vector z ij The jth element of ijk Represents vector z ij The kth element of , (0) represents the value of the element at the initial moment, i.e., t=0.

[0102] The estimated vector of the target agent in the target alliance is determined according to the first preset time gain operator and the directed unbalanced communication topology graph; the expression of the estimated vector is:

[0103]

[0104] Among them, zij is the estimated vector of the jth target agent in the i-th target coalition, is the update rate of the estimation vector of the jth target agent in the i-th target alliance, φ(t;∈T) is the first preset time gain operator with preset time ∈T, ∈T is the first preset time, 0<∈<1.

[0105] Applying this algorithm, we can ensure that z ij Converges to the Laplace matrix L at the first preset time ∈T i The left zero eigenvector ξ of i ,ξ i T L i =0ξ i T .

[0106] S6: Determine the average gradient estimation item of the target agent according to the estimation vector of the target agent; the average gradient estimation item converges to the average estimation value of the overall cost function of the target alliance to which the target agent belongs at a second preset time.

[0107] Among them, S6 specifically includes:

[0108] A second preset time gain operator is established; the expression of the second preset time gain operator is:

[0109]

[0110] in, is the second preset time gain operator, and φ(t; T) is the first preset time gain operator with the preset time being T.

[0111] Initialize auxiliary variables of the jth target agent in the i-th target coalition The kth element of and the average gradient estimate g ij The kth element of is 0, i∈{1,2,...,N}, N is the number of alliances participating in the game, j,k∈{1,2...,m i}.

[0112] According to the estimated vector of the target agent, the average gradient estimation term of the target agent is determined; the expression of the average gradient estimation term of the target agent is:

[0113]

[0114] in, f ij () is the cost function of the jth target agent in the i-th target alliance, y represents the set of decision state vectors of all target agents in all target alliances, η represents the set of Nash equilibrium search items of all target agents in all target alliances, η i represents the set of Nash equilibrium search items of all target agents in the i-th target alliance, N is the number of target alliances participating in the game, represents the mth target alliance in the i-th target alliance i The Nash equilibrium search items of the target agent, Indicates m i q-dimensional real vector, for The update rate, z ijj is the estimated vector z of the jth target agent in the i-th target alliance ij The jth element of is the second preset time gain operator, g ijk is the average gradient estimate g of the jth target agent in the i-th target alliance ij The kth element of .

[0115] S7: Determine the Nash equilibrium search item of the target intelligent agent according to the average gradient estimation item of the target intelligent agent; the Nash equilibrium search item converges to the Nash equilibrium solution of the target intelligent agent in the second preset time; the Nash equilibrium solution is the decision state vector that minimizes the overall cost function of the target alliance to which the target intelligent agent belongs.

[0116] Each agent has a corresponding cost function. The cost function of the jth agent in the i-th alliance is denoted as f ij (y i ,y -i ), where y i represents the sum of the decision variables of all agents participating in the game in the i-th alliance, that is, y -i represents the sum vector of decision variables of other alliances, that is,

[0117] The cost function f of the alliance i (y i ,y -i ) is the sum of the cost functions of each agent in the alliance, that is,

[0118] The desired Nash equilibrium solution (optimal decision state vector) is that the goal of each agent participating in the game is to minimize the overall cost function of the alliance to which it belongs by changing its own decision state.

[0119] If there exists an optimal decision state vector satisfy Then this optimal decision state vector is the Nash equilibrium solution. It can be seen that the Nash equilibrium solution is the optimal decision state that all alliances participating in the game and the intelligent agents within them should pursue.

[0120] Among them, S7 specifically includes:

[0121] Initialize the target agent’s Nash equilibrium search term to a zero vector.

[0122] According to the average gradient estimation term of the target agent, the Nash equilibrium search term of the target agent is determined; the expression of the Nash equilibrium search term of the target agent is:

[0123]

[0124] Among them, η ij is the Nash equilibrium search item of the jth target agent in the i-th target alliance, is η ij The update rate of g is, κ is the Nash equilibrium search rate control parameter, κ>0, g ijj is the average gradient estimate g of the jth target agent in the i-th target alliance ij The j-th element of .

[0125] S8: Controlling the agents in the unmanned swarm system based on the Nash equilibrium search terms of all target agents in all target alliances, including:

[0126] The first gain matrix and the second gain matrix are constructed according to the decision-making state dynamics model of multi-coalition game agents. The calculation formulas of the first gain matrix and the second gain matrix are as follows:

[0127]

[0128] Among them, K ij1 is the first gain matrix of the jth target agent in the i-th target alliance, K ij2 is the second gain matrix of the jth target agent in the i-th target alliance, I q is a q×q-dimensional identity matrix with all diagonal elements being 1 and all other elements being 0.

[0129] According to the Nash equilibrium search items of all target agents in all target alliances, the first gain matrix and the second gain matrix, the control input of all target agents in all target alliances is obtained; the formula of the control input of the target agent is:

[0130]

[0131] Among them, u' ij is the control input of the jth target agent in the i-th target coalition.

[0132] Control the agents in the unmanned swarm system based on the control inputs of all target agents in all target coalitions.

[0133] The present application proposes an agent control method for multi-alliance game for a multi-alliance game scenario where cooperation and gaming coexist. By introducing a preset time gain operator (a first gain matrix and a second gain matrix), the algorithm converges within a preset time. By designing and executing an estimation vector of a left zero eigenvector, the influence caused by a directed unbalanced communication topology graph is resolved. Subsequently, based on an average gradient estimation term, the limitation that the agent can only obtain its own cost function and needs to optimize the entire alliance cost function is resolved, and a Nash equilibrium search term is designed in combination with the estimation vector. Finally, a Nash equilibrium search term output adjustment control algorithm is designed, and based on the Nash equilibrium search term and its integral designed in the previous order, the agent (a high-order heterogeneous general linear agent) is driven to converge to a Nash equilibrium solution within a preset time.

[0134] Consider a connectivity preservation game involving three alliances, i.e., alliance number i∈{1,2,3}. There are 3 individuals in alliance 1, 4 individuals in alliance 2, and 4 individuals in alliance 3, i.e., m1=3, m2=4, m3=4. Their communication topology is as follows: Figure 2 shown.

[0135] In the game process, the decision variable of each individual is its position information, that is, y ij =[y xij ,y yij ] T , where y xij represents the horizontal coordinate of the jth agent in the ith alliance, y yij Represents the vertical coordinate of the jth agent in the ith alliance.

[0136] Considering that they participate in a multi-coalition game scenario where sensor connectivity is maintained, the cost function f of each agent is ij The cost function within the alliance Cost function between alliances Combined, the expression is as follows:

[0137]

[0138] Among them, the cost function expression within the alliance is:

[0139]

[0140] d1=[0,0] T m,d2=[5,5] T m,d3=[-5,-5] T m

[0141] ||·|| represents the norm operator, m is the unit (meter), d i is the bias vector.

[0142] The cost function expression between alliances is:

[0143]

[0144] Using the method of this application, the trajectory diagram of the decision variables of each intelligent agent and the estimated error curve diagram of the left zero eigenvector of the preset time directed graph are obtained as follows: Figure 3-Figure 4 As shown, Figure 3 It indicates that the agent searches for the trajectory of its Nash equilibrium point from the initial position by applying the algorithm, which shows that the agent can search for the Nash equilibrium solution within the preset time; Figure 4 The parameter in the upper right corner represents the error (norm) between the estimated vector and the true value of the left zero eigenvector. Figure 4 It can be seen that the estimation error of each agent for the left zero eigenvector is basically 0 after 0.5S, that is, the present application can effectively eliminate the influence of the directed unbalanced communication topology on the communication between agents.

[0145] Based on the same inventive concept, the embodiment of the present application also provides an agent control device for multi-alliance games for implementing the agent control method for multi-alliance games involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of one or more agent control devices for multi-alliance games provided below can refer to the limitations of the agent control method for multi-alliance games above, and will not be repeated here.

[0146] In an exemplary embodiment, Figure 5 As shown, an intelligent agent control device for a multi-coalition game is provided, comprising:

[0147] The alliance division module is used to divide the intelligent agents in the unmanned swarm system into multiple alliances; each alliance includes: multiple intelligent agents.

[0148] The directed unbalanced communication topology diagram establishment module is used to establish the directed unbalanced communication topology diagram of the unmanned cluster system based on the actual communication situation between each intelligent agent in the unmanned cluster system.

[0149] The model building module is used to establish a decision-making state dynamics model of multi-alliance game agents based on the agents in all alliances.

[0150] The target alliance determination module is used to determine the target alliance according to the decision state dynamics model of multi-alliance game agents; the target alliance is an alliance with agents that meet the trigger conditions; the agent that meets the trigger conditions in the target alliance is the target agent.

[0151] The estimated vector determination module is used to determine the Laplace matrix of the target alliance and the estimated vector of the target intelligent agent in the target alliance according to the directed unbalanced communication topology graph; the estimated vector converges to the left zero eigenvector of the Laplace matrix of the target alliance to which the target intelligent agent belongs at a first preset time.

[0152] The average gradient estimation item determination module is used to determine the average gradient estimation item of the target intelligent agent according to the estimation vector of the target intelligent agent; the average gradient estimation item converges to the average estimation value of the overall cost function of the target alliance to which the target intelligent agent belongs within a second preset time.

[0153] The Nash equilibrium search item determination module is used to determine the Nash equilibrium search item of the target intelligent agent according to the average gradient estimation item of the target intelligent agent; the Nash equilibrium search item converges to the Nash equilibrium solution of the target intelligent agent within a second preset time; the Nash equilibrium solution is the decision state vector that minimizes the overall cost function of the target alliance to which the target intelligent agent belongs.

[0154] A control module is used to control the agents in the unmanned swarm system according to the Nash equilibrium search terms of all target agents in all target coalitions.

[0155] The main feature of this application is to design and use the estimation of the left zero eigenvector to eliminate the interference caused by the unbalanced directed graph, and to complete the preset time multi-coalition game Nash equilibrium search of high-order heterogeneous intelligent agents by designing the average gradient estimation term, Nash equilibrium search term and output adjustment term that converge within the preset time.

[0156] This application focuses on the multi-alliance game scenarios of intra-alliance cooperation and inter-alliance games under unbalanced directed communication topology graphs, and provides a solution for participating intelligent agents to search for Nash equilibrium solutions within any preset time. The application can effectively improve the efficiency of all intelligent agent states in multi-alliance games converging to Nash equilibrium solutions, support intelligent agents participating in multi-alliance games in multi-alliance game scenarios such as multi-group connectivity preservation games, integrate game-related information within a preset time and converge to the Nash equilibrium solution site where connectivity is preserved, and improve the rate at which intelligent agents search for the Nash equilibrium solution site where connectivity is preserved.

[0157] Using this method, high-order heterogeneous general linear agents can be controlled to search for Nash equilibrium solutions of multi-coalition games by interacting in unbalanced directed graphs when only knowing their own cost functions. This method can be applied to multiple scenarios such as multi-cluster unmanned system connectivity maintenance and multi-coalition congestion games to control agents to search for Nash equilibrium solutions within a preset time.

[0158] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the intelligent agent control data of the multi-alliance game. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for controlling an intelligent agent in a multi-alliance game is implemented.

[0159] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0160] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0161] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0162] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0163] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0164] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for controlling an intelligent agent in a multi-coalition game, characterized in that: The agent control method of the multi-coalition game includes: The intelligent agents in the unmanned swarm system are divided into a plurality of alliances; each of the alliances comprises: a plurality of intelligent agents; Based on the actual communication situation between each intelligent agent in the unmanned swarm system, a directed unbalanced communication topology diagram of the unmanned swarm system is established; Establish a decision-making state dynamics model of multi-coalition game agents based on all agents in the alliance; Determine the target alliance according to the decision state dynamics model of the multi-alliance game agent; the target alliance is an alliance with agents that meet the trigger conditions; the agent that meets the trigger conditions in the target alliance is the target agent; According to the directed unbalanced communication topology graph, determining the Laplacian matrix of the target alliance and the estimated vector of the target agent in the target alliance; the estimated vector converges to the left zero eigenvector of the Laplacian matrix of the target alliance to which the target agent belongs at a first preset time; Determining an average gradient estimation term of the target agent according to the estimation vector of the target agent; wherein the average gradient estimation term converges to an average estimation value of the overall cost function of the target alliance to which the target agent belongs within a second preset time; Determining a Nash equilibrium search item for the target intelligent agent according to the average gradient estimation item of the target intelligent agent; the Nash equilibrium search item converges to a Nash equilibrium solution of the target intelligent agent within the second preset time; the Nash equilibrium solution is a decision state vector that minimizes the overall cost function of the target alliance to which the target intelligent agent belongs; The agents in the unmanned swarm system are controlled according to the Nash equilibrium search items of all target agents in all target coalitions.

2. The agent control method of multi-coalition game according to claim 1, characterized in that: The expression of the decision-making state dynamics model of the multi-coalition game agent is: y ij =C ij x ij ; Among them, y ij represents the decision state vector of the jth agent in the i-th alliance, y ij ∈R q , R q represents a q-dimensional real vector, u ij represents the control input of the jth agent in the ith coalition, Indicates p ij dimensional real vector, x ij represents the internal state of the jth agent in the ith alliance, Indicates n ij dimensional real vector, represents the update rate of the internal state of the jth agent in the ith alliance, A ij represents the system dynamics matrix of the jth agent in the ith alliance, B ij represents the system input matrix of the jth agent in the ith alliance, C ij represents the decision matrix of the jth agent in the ith coalition.

3. The agent control method of multi-coalition game according to claim 2, characterized in that: Based on the actual communication situation between each intelligent agent in the unmanned swarm system, a directed unbalanced communication topology diagram of the unmanned swarm system is established, including: Based on the actual communication between the agents in the unmanned cluster system, the communication topology within the i-th alliance is established. in, represents the set of all agents participating in the i-th alliance, m i represents the number of agents in the ith alliance, represents the first agent in the ith alliance, Indicates the mth in the ith alliance i Agents; E i represents the communication edge set of agents in the ith alliance, represents a communication edge from the jth agent in the ith alliance to the kth agent in the ith alliance, represents the jth agent in the i-th alliance, represents the kth agent in the i-th alliance; The weighted adjacency matrix of the ith alliance is established according to the communication topology within the ith alliance. The weighted adjacency matrix is ​​a square matrix. When , the value of the element in the kth row and jth column of the weight adjacency matrix of the i-th alliance is otherwise According to the weighted adjacency matrix of the ith alliance, determine the in-degree of the jth agent in the ith alliance and the out-degree of the jth agent in the ith alliance; the in-degree of the jth agent in the ith alliance Represents the value of the element in the jth row and kth column of the weighted adjacency matrix of the i-th alliance, and the out-degree of the j-th agent in the i-th alliance The directed unbalanced communication topology graph of the unmanned cluster is constructed by the in-degree of all agents in the i-th alliance and the out-degree of all agents.

4. The agent control method of multi-coalition game according to claim 3, characterized in that: Determining the Laplacian matrix of the target alliance and the estimated vector of the target agent in the target alliance according to the directed unbalanced communication topology graph specifically includes: Determine the Laplace matrix of the target alliance; the expression of the Laplace matrix is: Among them, L i is the Laplacian matrix of the i-th target alliance, D i is a diagonal matrix constructed by the out-degrees of all agents in the i-th target alliance, is the weighted adjacency matrix of the ith coalition; A first preset time gain operator is established; the expression of the first preset time gain operator is: Wherein, h>1, h is the gain rate control parameter of the first preset time gain operator, T is the second preset time, φ(t;T) is the first preset time gain operator with the preset time T, and t is time; The estimated vector of the target agent in the target alliance is determined according to the first preset time gain operator and the directed unbalanced communication topology graph; the expression of the estimated vector is: Among them, z ij is the estimated vector of the jth target agent in the i-th target coalition, is the update rate of the estimation vector of the jth target agent in the i-th target alliance, φ(t;∈T) is the first preset time gain operator with preset time ∈T, ∈T is the first preset time, 0<∈<1.

5. The agent control method of multi-coalition game according to claim 4, characterized in that: Determining the average gradient estimation item of the target agent according to the estimation vector of the target agent includes: A second preset time gain operator is established; the expression of the second preset time gain operator is: in, is the second preset time gain operator, φ(t; T) is the first preset time gain operator with the preset time being T; Initialize auxiliary variables of the jth target agent in the i-th target coalition The kth element of and the average gradient estimate g ij The kth element of is 0, i∈{1,2,…,N}, N is the number of alliances participating in the game, j,k∈{1,2…,m i }; According to the estimation vector of the target agent, the average gradient estimation term of the target agent is determined; the expression of the average gradient estimation term of the target agent is: in, f ij () is the cost function of the jth target agent in the i-th target alliance, y represents the set of decision state vectors of all target agents in all target alliances, η represents the set of Nash equilibrium search items of all target agents in all target alliances, η i represents the set of Nash equilibrium search items of all target agents in the i-th target alliance, N is the number of target alliances participating in the game, represents the mth target in the i-th target alliance i The Nash equilibrium search items of the target agent, Indicates m i q-dimensional real vector, for The update rate, z ijj is the estimated vector z of the jth target agent in the i-th target alliance ij The jth element of is the second preset time gain operator, g ijk is the average gradient estimate g of the jth target agent in the i-th target alliance ij The kth element of .

6. The agent control method of multi-coalition game according to claim 5, characterized in that: Determining a Nash equilibrium search term of the target intelligent agent according to the average gradient estimation term of the target intelligent agent includes: Initializing the Nash equilibrium search item of the target agent to a zero vector; According to the average gradient estimation item of the target agent, the Nash equilibrium search item of the target agent is determined; the expression of the Nash equilibrium search item of the target agent is: Among them, η ij is the Nash equilibrium search item of the jth target agent in the i-th target alliance, is η ij The update rate of g is, κ is the Nash equilibrium search rate control parameter, κ>0, g ijj is the average gradient estimate g of the jth target agent in the i-th target alliance ij The j-th element of .

7. The agent control method of multi-coalition game according to claim 6, characterized in that: Controlling the agents in the unmanned swarm system according to the Nash equilibrium search items of all target agents in all target alliances specifically includes: The first gain matrix and the second gain matrix are constructed according to the decision-making state dynamics model of the multi-coalition game agent; the calculation formulas of the first gain matrix and the second gain matrix are respectively: Among them, K ij1 is the first gain matrix of the jth target agent in the i-th target alliance, K ij2 is the second gain matrix of the jth target agent in the i-th target alliance, I q is a q×q-dimensional identity matrix with all diagonal elements being 1 and all other elements being 0; According to the Nash equilibrium search items of all target agents in all target alliances, the first gain matrix and the second gain matrix, the control input of all target agents in all target alliances is obtained; the formula of the control input of the target agent is: Among them, u' ij is the control input of the jth target agent in the i-th target coalition; The agents in the unmanned swarm system are controlled according to the control inputs of all target agents in all target alliances.

8. An intelligent agent control device for multi-coalition game, characterized in that: The agent control device of the multi-coalition game comprises: The alliance division module is used to divide the intelligent agents in the unmanned cluster system into multiple alliances; each of the alliances includes: multiple intelligent agents; A directed unbalanced communication topology diagram establishment module is used to establish a directed unbalanced communication topology diagram of the unmanned cluster system based on the actual communication conditions between various intelligent agents in the unmanned cluster system; A model building module is used to build a decision-making state dynamics model of multi-alliance game agents based on the agents in all alliances; A target alliance determination module is used to determine a target alliance according to the decision state dynamics model of the multi-alliance game agent; the target alliance is an alliance with agents that meet the trigger conditions; the agent in the target alliance that meets the trigger conditions is the target agent; An estimation vector determination module is used to determine the Laplace matrix of the target alliance and the estimation vector of the target agent in the target alliance according to the directed unbalanced communication topology graph; the estimation vector converges to the left zero eigenvector of the Laplace matrix of the target alliance to which the target agent belongs at a first preset time; an average gradient estimation item determination module, configured to determine an average gradient estimation item of the target agent according to the estimation vector of the target agent; the average gradient estimation item converges to an average estimation value of the overall cost function of the target alliance to which the target agent belongs within a second preset time; A Nash equilibrium search item determination module, used to determine the Nash equilibrium search item of the target intelligent agent according to the average gradient estimation item of the target intelligent agent; the Nash equilibrium search item converges to the Nash equilibrium solution of the target intelligent agent within the second preset time; the Nash equilibrium solution is a decision state vector that minimizes the overall cost function of the target alliance to which the target intelligent agent belongs; A control module is used to control the agents in the unmanned cluster system according to the Nash equilibrium search items of all target agents in all target alliances.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the agent control method for a multi-alliance game according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the intelligent agent control method for a multi-coalition game according to any one of claims 1 to 7 is implemented.

Images