Intelligent agent interaction information aggregation method, system and device based on global state approximation, medium and product

Abstract

The invention discloses an agent interaction information aggregation method, system and device based on global state approximation, a medium and a product, and relates to the technical field of computers.The method comprises the steps that a current hierarchical collaborative collection task scene is determined, and the current hierarchical collaborative collection task scene is initialized; then, taking interaction information generated by a current agent and an interaction information set transmitted by a teammate agent in the initialized current hierarchical collaborative collection task scene as target data, and preprocessing the target data; and inputting the preprocessed target data into the agent interaction information aggregation model to aggregate the agent interaction information, and outputting an agent interaction information aggregation result prediction value. The accuracy of interaction information aggregation and the utilization rate of historical information can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, system, device, medium and product for aggregating intelligent agent interaction information based on global state approximation. Background Art

[0002] Cooperative multi-agent reinforcement learning has attracted widespread attention in the field of artificial intelligence and has shown its potential to solve practical problems in many fields, such as autonomous vehicle swarms, traffic signal control, and intelligent warehousing systems. In some observable environments, insufficient understanding of the global scene and the task execution status of other agents may lead to the failure of collaboration. In this case, through information exchange between agents, the agents can have a more complete understanding of the global state and optimize the making of collaborative decisions. In the process of information exchange, it is an important challenge for agents to remove redundancy, filter key information, and dynamically fuse various types of information received from different sources to form an effective aggregate representation.

[0003] However, at each time step, the agent receives interactive information shared by teammates of different roles and states, and it is difficult to accurately evaluate the relevance and importance of this information to the agent's strategy. Existing interactive information aggregation methods, including concatenation, weighting, averaging, and attention aggregation of the original interactive information vector, enable the agent to autonomously learn the importance weights of different messages received at the same time step. However, these aggregation methods ignore the differences in messages from different sources, making it difficult to accurately aggregate messages from different sources, and insufficient use of historical messages, making the aggregated representation unable to reflect the global scene state and unable to provide a global perception basis for the subsequent learning of collaborative strategies. Summary of the invention

[0004] The purpose of this application is to provide a method, system, device, medium and product for aggregating agent interaction information based on global state approximation, which can solve the problem that it is difficult to accurately aggregate messages from different sources and the low utilization rate of historical information in the agent interaction information aggregation method, resulting in the aggregation representation being unable to reflect the global scene state.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for aggregating agent interaction information based on global state approximation, comprising:

[0007] Determine a current hierarchical collaborative collection task scenario and initialize the current hierarchical collaborative collection task scenario; the hierarchical collaborative collection task scenario includes multiple agents and multiple targets;

[0008] Acquire target data; the target data includes interaction information generated by the current agent itself in the initialized current hierarchical collaborative collection task scene and a set of interaction information transmitted by teammate agents; the teammate agents are agents other than the current agent in the initialized hierarchical collaborative collection task scene;

[0009] Preprocessing the target data to obtain preprocessed target data;

[0010] The preprocessed target data is input into the agent interaction information aggregation model to aggregate the agent interaction information, and the predicted value of the agent interaction information aggregation result is output; the agent interaction information aggregation model is obtained by training the neural network model with training samples; the training samples include the interaction information samples generated by the current agent itself, the interaction information set samples transmitted by the teammate agent, and the agent interaction information aggregation result sample value;

[0011] Among them, the agent interaction information aggregation model includes a graph convolutional network, a gated recurrent unit and an encoder connected in sequence.

[0012] Optionally, the current hierarchical collaborative collection task scenario is initialized, specifically including:

[0013] The grid size, number of agents, and number of targets of the current hierarchical collaborative collection task scenario are initialized, and the maximum number of steps for the agent to perform actions in the hierarchical collaborative collection task scenario is set to obtain the initialized hierarchical collaborative collection task scenario.

[0014] Optionally, preprocessing the target data to obtain preprocessed target data specifically includes:

[0015] Construct a feature matrix based on the interaction information generated by the current agent and the set of interaction information transmitted by teammate agents;

[0016] The interaction information generated by the current agent itself and the set of interaction information transmitted by the teammate agent are input into the attention mechanism network, and the attention weight of each agent to each interaction information is output. An adjacency matrix is ​​constructed according to the attention weight of each agent to each interaction information, and the feature matrix and the adjacency matrix are used as preprocessed target data.

[0017] Optionally, the training process of the agent interaction information aggregation model specifically includes:

[0018] Construct training samples based on the interaction information samples generated by the current agent and the interaction information set samples transmitted by teammate agents;

[0019] Inputting the training sample into the graph convolutional network to obtain a graph representation at the current moment;

[0020] Obtaining historical graph representations of the previous k moments, and inputting the historical graph representations of the previous k moments and the graph representation of the current moment into the gated recurrent unit to obtain a historical time series representation;

[0021] Inputting the current graph representation and the historical time series representation into the encoder to obtain an aggregate representation;

[0022] Inputting the aggregate representation into a decoder to obtain a decoded output of the aggregate representation;

[0023] The global scene state sample value of the initialized hierarchical collaborative collection task scene is obtained, and the loss value is calculated based on the decoded output of the aggregate representation and the global scene state sample value, and the graph convolutional network parameters, gated recurrent unit parameters, encoder parameters, and decoder parameters are iteratively updated according to the loss value until the loss value is less than a preset threshold or the number of iterations reaches a maximum value, then the iterative update is stopped, and the trained graph convolutional network, gated recurrent unit, and encoder are used as an agent interaction information aggregation model.

[0024] Optionally, a mean square error loss function is used to calculate the loss value, and the expression of the mean square error loss function is:

[0025]

[0026] in, represents the loss value; μ represents the encoder parameter; η represents the decoder parameter; Express expectation; t Represents the global state of the hierarchical collaborative collection scenario; represents the global aggregate representation; represents the decoded output of the global aggregate representation; s t Represents the global scene state of the hierarchical collaborative collection task scene after initialization; Represents the square of the L2 norm.

[0027] In a second aspect, the present application provides an agent interaction information aggregation system based on global state approximation, wherein the agent interaction information aggregation system based on global state approximation is based on the agent interaction information aggregation method based on global state approximation, and the agent interaction information aggregation system based on global state approximation includes:

[0028] A scene determination and initialization unit, used to determine the current hierarchical collaborative collection task scene and initialize the current hierarchical collaborative collection task scene; the hierarchical collaborative collection task scene includes multiple agents and multiple targets;

[0029] A target data acquisition unit, used to acquire target data; the target data includes interaction information generated by the current agent itself in the initialized current hierarchical collaborative collection task scene and a set of interaction information transmitted by teammate agents; the teammate agents are agents other than the current agent in the initialized hierarchical collaborative collection task scene;

[0030] A preprocessing unit, used for preprocessing the target data to obtain preprocessed target data;

[0031] An agent interaction information aggregation result prediction unit is used to input the preprocessed target data into the agent interaction information aggregation model to aggregate the agent interaction information and output the agent interaction information aggregation result prediction value; the agent interaction information aggregation model is obtained by training the neural network model with training samples; the training samples include the interaction information samples generated by the current agent itself, the interaction information set samples transmitted by the teammate agents and the agent interaction information aggregation result sample values;

[0032] Among them, the agent interaction information aggregation model includes a graph convolutional network, a gated recurrent unit and an encoder connected in sequence.

[0033] Optionally, the preprocessing unit includes:

[0034] The feature matrix construction subunit is used to construct a feature matrix based on the interaction information generated by the current agent itself and the set of interaction information transmitted by the teammate agents;

[0035] The adjacency matrix construction subunit is used to input the interaction information generated by the current intelligent agent itself and the set of interaction information transmitted by the teammate intelligent agent into the attention mechanism network, output the attention weight of each intelligent agent to each interaction information, and construct the adjacency matrix according to the attention weight of each intelligent agent to each interaction information, and use the feature matrix and the adjacency matrix as the preprocessed target data.

[0036] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-mentioned methods for aggregating intelligent agent interaction information based on global state approximation.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned methods for aggregating agent interaction information based on global state approximation.

[0038] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for aggregating agent interaction information based on global state approximation.

[0039] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0040] The present application provides a method, system, device, medium and product for aggregating agent interaction information based on global state approximation. Agent interaction information is aggregated through an agent interaction information aggregation model. The agent interaction information aggregation model includes a graph convolutional network, a gated recurrent unit and an encoder connected in sequence. In the process of interaction information aggregation, the graph convolutional network is used to extract spatial graph features (i.e., graph representation) between agents, which can more accurately evaluate the importance of each interaction information and better understand the complex interaction relationship between agents, thereby improving the accuracy of interaction information aggregation; by using a gated recurrent unit to extract the time correlation between historical interaction information and considering the evolution of information in a time series, not only can the aggregated representation cover a longer amount of information, but also improve the utilization rate of information. Compared with the use-and-discard method of most existing single-step interactions, the combination of historical interaction information can more flexibly capture dynamic changes in time spans; using an encoder and a decoder, the aggregated representation is approximated to the global scene state, further improving the accuracy of interaction information fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 This is an application environment diagram of an interactive information aggregation method based on global state approximation in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of a flow chart of a method for aggregating interactive information based on global state approximation provided in one embodiment of the present application;

[0044] Figure 3 A schematic diagram of the training process of an agent interaction information aggregation model provided in another embodiment of the present application;

[0045] Figure 4Another embodiment of the present application provides a schematic diagram of the position change of the intelligent agent and the target during the collection process in a simulation experiment scenario of a hierarchical collaborative collection task; wherein (a) to (f) are schematic diagrams of the position change of the intelligent agent and the target during the collection process in the hierarchical collaborative collection task;

[0046] Figure 5 A schematic diagram of functional modules of an interactive information aggregation system based on global state approximation provided by another embodiment of the present application;

[0047] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application.

[0048] Reference numerals:

[0049] Terminal-102, server-104, scene determination and initialization unit-1, target data acquisition unit-2, preprocessing unit-3, intelligent agent interaction information aggregation result prediction unit-4. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] To solve the above problems, this application proposes a method, system, device, medium and product for aggregating interactive information based on global state approximation to optimize the information interaction process between multiple agents. This application aims to enable the agent to evaluate the importance of the received interactive information from the perspective of time and space, and dynamically generate an aggregated representation that can approximate the global collaborative state, providing complete perceptual input for collaborative strategy generation. First, a graph convolutional network is used to extract spatial graph features between agents to more accurately evaluate the importance of each interactive information; secondly, a gated recurrent unit is used to extract the temporal correlation between historical interactive information, so that the aggregated representation is rich in information with a longer time span, and the utilization rate of the interactive information can be improved; finally, the encoder and decoder principles are used to approximate the aggregated representation to the global scene state, thereby improving the accuracy of interactive information fusion.

[0052] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The agent interaction information aggregation method based on global state approximation provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the target data to the server 104. After the server 104 receives the target data, for the target data, the server 104 determines the current hierarchical collaborative collection task scenario and initializes the current hierarchical collaborative collection task scenario; obtains the target data; preprocesses the target data to obtain the preprocessed target data; inputs the preprocessed target data into the intelligent agent interaction information aggregation model to aggregate the intelligent agent interaction information, and outputs the intelligent agent interaction information aggregation result prediction value. The server 104 can feedback the obtained intelligent agent interaction information aggregation result prediction value to the terminal 102. In addition, in some embodiments, the method for aggregating agent interaction information based on global state approximation can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform agent interaction information aggregation based on global state approximation on the target data, or the server 104 can obtain the target data from the data storage system and perform agent interaction information aggregation based on global state approximation on the target data.

[0054] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0055] In an exemplary embodiment, Figure 2 As shown, a method for aggregating agent interaction information based on global state approximation is provided. The method is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps S1 to S4. Among them:

[0056] Step S1, determine the current hierarchical collaborative collection task scenario, and initialize the current hierarchical collaborative collection task scenario; the hierarchical collaborative collection task scenario includes multiple agents and multiple targets.

[0057] As an optional implementation, the current hierarchical collaborative collection task scenario is initialized, specifically including:

[0058] The grid size, number of agents, and number of targets of the current hierarchical collaborative collection task scenario are initialized, and the maximum number of steps for the agent to perform actions in the hierarchical collaborative collection task scenario is set to obtain the initialized hierarchical collaborative collection task scenario.

[0059] Specifically, the grid size of the current hierarchical collaborative collection task scene is initialized to G×G, the number of agents is N, the number of targets is F, the local perception of the agent is the S×S grid area centered on itself, the maximum number of iterative training rounds is set to TRAIN_NUM, and the maximum number of steps that the agent can take in the scene is set to NUM_STEPS.

[0060] Step S2, obtaining target data; the target data includes the interaction information generated by the current agent itself in the initialized current hierarchical collaborative collection task scene and the interaction information set transmitted by the teammate agent; the teammate agent is the agent other than the current agent in the initialized hierarchical collaborative collection task scene. The current agent and the teammate agent form an agent team.

[0061] Specifically, step S21 traverses the number of steps NUM_STEPS that the agent team performs in the current hierarchical collaborative collection task scenario after initialization.

[0062] Step S22, traverse the number N of agents in the agent team.

[0063] Step S23: for each agent, obtain the set of interaction information transmitted by the teammate agent in, represents the set of interactive information received by the ith agent, represents the jth interactive information vector received at time t, and C represents the total number of interactive information; at the same time, the interactive information generated by the current agent itself is obtained The interaction information generated by the current agent itself And the set of interactive information transmitted by teammates As the target data. The mutual information specifically refers to the (p x ,p y ,l), where p x ,p yRepresents the two-dimensional grid coordinates, l represents the level, and only when the sum of the levels of a single agent / multiple aggregated agents is higher than or equal to the target level can the target be collected, otherwise the target cannot be collected. For example, multiple unmanned boats go to salvage marine garbage and take the corresponding marine garbage as the target. Only when the sum of the levels of multiple unmanned boats or the level of a single unmanned boat is greater than the level of the target marine garbage, the unmanned boat team can have the ability to collect marine garbage, otherwise, the target collection cannot be completed.

[0064] Step S3, preprocessing the target data to obtain preprocessed target data.

[0065] As an optional implementation manner, preprocessing the target data to obtain preprocessed target data specifically includes:

[0066] Step S31, constructing a feature matrix based on the interaction information generated by the current agent itself and the set of interaction information transmitted by teammate agents.

[0067] Step S32, input the interaction information generated by the current agent itself and the set of interaction information transmitted by the teammate agent into the attention mechanism network, output the attention weight of each agent to each interaction information, and construct an attention matrix based on the attention weight of each agent to each interaction information, and use the attention matrix as the adjacency matrix, and use the feature matrix and the adjacency matrix as the preprocessed target data.

[0068] Specifically, the interactive information generated by the current agent itself And the set of interactive information transmitted by teammates Composition feature matrix The interaction information generated by the current agent itself And the set of interactive information transmitted by teammates Input into the attention mechanism network, and output the attention weight α of each agent i on each interactive information ij , based on the attention weight α of each agent i on each interactive information ij Construct the adjacency matrix A, that is, the adjacency matrix A ij =α ij .

[0069] Step S4, input the preprocessed target data into the agent interaction information aggregation model to aggregate the agent interaction information, and output the predicted value of the agent interaction information aggregation result; the agent interaction information aggregation model is obtained by training the neural network model with training samples; the training samples include the interaction information samples generated by the current agent itself, the interaction information set samples transmitted by the teammate agent, and the agent interaction information aggregation result sample value. The agent interaction information aggregation model includes a graph convolutional network, a gated recurrent unit, and an encoder connected in sequence.

[0070] As an optional implementation, Figure 3 As shown, the training process of the agent interaction information aggregation model in step S4 specifically includes:

[0071] Step S41, construct training samples based on the interaction information samples generated by the current agent itself and the interaction information set samples transmitted by the teammate agents; wherein, the interaction information samples generated by the current agent itself A sample set of interactive information transmitted with teammate agents Preprocess and get the feature matrix and the adjacency matrix A ij =α ij , the adjacency matrix is Figure 3 The attention matrix in , the interactive information sample generated by the current agent itself A sample set of interactive information transmitted with teammate agents That is Figure 3 The interactive information 1, interactive information 2, ... interactive information C in the .

[0072] Step S42, input the training sample into the graph convolution network to obtain the graph representation at the current moment; wherein, the feature matrix E and the adjacency matrix A are input into the graph convolution network to obtain the graph representation at the current moment And the generated graph representation Stored in memory cell d, Assuming that the current time step is the 4th time step, the graph representation is completed After the storage of , the memory unit currently contains 4 graph representations of agent i, namely These stored graph representations will serve as input to subsequent gated recurrent units.

[0073] Step S43, obtaining the historical graph representation of the previous k moments, and inputting the historical graph representation of the previous k moments and the graph representation of the current moment into the gated recurrent unit to obtain the historical time series representation; wherein, the historical graph representation of the previous K moments is extracted from the memory unit D The historical graph representation and the current graph representation are input into the gated recurrent unit to obtain the historical time series representation GRU stands for Gated Recurrent Unit.

[0074] For example, when k=3, the historical graph representation of the previous k=3 time steps is combined and the gated recurrent unit is used to generate the historical temporal representation. The specific process is:

[0075] From memory unit Extract the historical graph representation of agent i for the first k = 3 time steps from Assuming that the current time step is the 4th time step (t=4), the historical graph representation of the 3rd time step (t=3) is extracted. History graph representation of the second time step (t=2) History graph representation of the first time step (t=1) Then the historical graph representation and the current graph representation are input into the gated recurrent unit together to output the historical temporal representation Assuming that the current time is the fourth time step (t=4), the history graph of the previous three moments is represented as and the current time graph representation Input to the gated recurrent unit, output historical timing representation

[0076] Step S44, inputting the current graph representation and the historical time series representation into the encoder to obtain an aggregate representation; Among them, Encoder represents an encoder.

[0077] Step S45, inputting the aggregate representation into a decoder to obtain a decoded output of the aggregate representation; wherein, Represents the decoded output of the aggregate representation, and Decoder represents the decoder.

[0078] Step S46, obtain the global scene state sample value of the initialized hierarchical collaborative collection task scene, calculate the loss value based on the decoded output of the aggregate representation and the global scene state sample value, and iteratively update the graph convolutional network parameters, gated recurrent unit parameters, encoder parameters, and decoder parameters according to the loss value, until the loss value is less than a preset threshold or the number of iterations reaches a maximum value, then stop the iterative update, and use the trained graph convolutional network, gated recurrent unit, and encoder as an agent interaction information aggregation model.

[0079] Among them, it is determined whether the current number of iterations is greater than the preset maximum number of iterations. If so, the iterative update is stopped; or, it is determined whether the current loss value is less than a preset threshold. If so, the iterative update is stopped, otherwise, the steps S41 to S46 are continued to be looped to update each network parameter.

[0080] Before training the agent interaction information aggregation model, it also includes: initializing graph convolutional network parameters, gated recurrent unit parameters, encoder parameters and decoder parameters.

[0081] As an optional implementation, a mean square error loss function is used to calculate the loss value, and the expression of the mean square error loss function is:

[0082]

[0083] in, represents the loss value; μ represents the encoder parameter; η represents the decoder parameter; Express expectation; t Represents the global state of the hierarchical collaborative collection scenario; represents the global aggregate representation; represents the decoded output of the global aggregate representation; s t Represents the global scene state of the hierarchical collaborative collection task scene after initialization; Represents the square of the L2 norm, which is used to measure the difference between the decoded data and the original data, where the L2 norm is the Euclidean distance calculation.

[0084] Beneficial effects of this application:

[0085] 1) In the process of interactive information aggregation, the graph convolutional network is used to extract the spatial graph features between agents to more accurately evaluate the importance of each interactive information. In the process of interactive information aggregation, the graph convolutional network is used to extract the spatial graph features between agents. Since this feature contains the geometric topological structure between the current agent and the agent that sends the interactive information, different topological structures reflect the current degree of cooperation between each other. For example, for agents with closer spatial distances, higher importance weights should be given. Therefore, considering this feature is conducive to the agent to more accurately evaluate the importance of each interactive information.

[0086] 2) To address the problem of low utilization of historical information and to further enrich the aggregate representation, gated recurrent units are used to extract the temporal correlation between historical interaction information and consider the evolution of information in time series, which not only enables the aggregate representation to cover a longer period of time, but also improves the utilization of information. Compared with the use-and-forget approach of most existing single-step interactions, combining historical interaction information can more flexibly capture dynamic changes in time spans.

[0087] 3) In order to solve the problem that the aggregate representation cannot reflect the global scene state, an encoder-decoder architecture is used to map the aggregate representation to the global scene state to improve the accuracy of interactive information fusion. Compared with the existing methods, adding this encoding and decoding constraint can obtain an aggregate representation that is closer to the global state. Based on this aggregate representation, the observation-constrained intelligent agent can learn a more effective collaboration strategy, allowing the intelligent agent to understand the global task scenario through information interaction under the condition of local observation, collaborate with each other more efficiently, complete the hierarchical collaborative collection task, and improve the accuracy of interactive information fusion.

[0088] 4) Compared with other multi-agent information interaction methods, especially compared with the broadcast interaction method, during the interaction process, according to different scene states, it actively selects teammates who can currently provide interaction value to itself as interaction objects, shares local observation information representation, and uses targeted interaction methods to improve the efficiency of interaction, obtain more streamlined interaction information, and reduce the complexity of collaborative strategy input; through the selection of interaction objects, the communication redundancy and communication overhead of the system can be reduced, the decision delay caused by a large amount of message transmission can be avoided, and it can be easier to deploy in the real world.

[0089] The following is a further detailed description of the agent interaction information aggregation method based on global state approximation in conjunction with another exemplary embodiment. Figure 2 The specific implementation steps of the process include the following steps:

[0090] Step 1: Determine the current hierarchical collaborative collection task scenario, initialize the current hierarchical collaborative collection task scenario, and initialize all network parameters. The specific process is as follows:

[0091] Step 1.1 Initialize the "Hierarchical Collaborative Collection" task scenario:

[0092] The initial two-dimensional scene grid size is 11×11, the number of agents is 6, and each agent is randomly assigned a level. The level in this instantiated scene is 1 or 2. The number of targets in the scene is 4, and each target is randomly assigned a level, with a level of 1 or 2. The local perception of the agent is a 3×3 grid area centered on itself. The maximum number of iterations TRAIN_NUM=200000 is set, and the maximum number of time steps for the agent to perform actions in the scene is set NUM_STEPS=25, that is, the agent makes 25 consecutive decisions in the task scene. The goal of this task is that the 6 agents complete the collection of 4 targets at the fastest speed within 25 consecutive decisions. The collection is successful under the following conditions, that is, the collection can only be successful when the level of only one agent is equal to or higher than the target level. Assuming that the level of the agent is 1 and the level of the target is 2, the agent needs to cooperate with another teammate agent with a level of 1 or 2 to surround the target with a level of 2 in order to successfully collect it.

[0093] Step 1.2 Initialize the graph convolutional network parameters, gated recurrent unit parameters, encoder parameters, and decoder parameters.

[0094] Step 2: Obtain target data; the target data includes the interaction information generated by the current agent itself in the current hierarchical collaborative collection task scenario after initialization and the interaction information set transmitted by the teammate agent. The specific process is as follows:

[0095] Step 2.1 traverses the number of time steps NUM_STEPS = 25 that the agent team takes to perform actions in the scene.

[0096] Step 2.2 For each time step, traverse the number of agents in the team N = 6.

[0097] Step 2.3 For each agent i, at the tth time step, an interactive information vector is generated The content of the mutual information vector is specifically the local perception information of agent i, that is, the two-dimensional position information and level information of all targets and agents (including agent i itself) in the 3×3 grid area centered on the grid position of agent i: (p x ,p y ,l), where p x ,p y represents the two-dimensional grid coordinates, and l represents the level. Assuming that in the current instance, there is a teammate agent and a target within the perception range of teammate agent i, then the interaction information vector generated by agent i is in Indicates the location and level of the target, represents the position and level of agent i itself, Indicates the position and level of another teammate agent within the sensing range.

[0098] Step 2.4 After clarifying the interactive information vector generated by each agent, each agent i, at the tth time step, will also receive the interactive information set transmitted by the other five teammate agents, expressed as represents the interactive information vector transmitted by the jth teammate, the interactive information vector The content is the same as that defined in 2.3, specifically the local perception information of agent j, that is, the two-dimensional position information and level information of all targets and agents (including agent j itself) in the 3×3 grid area centered on the grid position of agent j.

[0099] Step S3, preprocessing the target data to obtain preprocessed target data.

[0100] Step 3.1 The interactive information generated by agent i itself And the set of interactive information received by agent i Composition feature matrix It is then used as the input of the graph convolutional network.

[0101] Step 3.2: The interactive information generated by agent i And the set of interactive information received by agent i As the input of the attention mechanism network, output the attention weight α of agent i on each interactive information ij , the attention weight α ij The adjacency matrix A will be used as the input of the graph convolutional network.

[0102] Step 4: Input the preprocessed target data (feature matrix E and adjacency matrix A) into the agent interaction information aggregation model to aggregate the agent interaction information and output the predicted value of the agent interaction information aggregation result.

[0103] Experimental verification and results:

[0104] 1. Scene determination and initialization:

[0105] 1) The simulation experiment environment is the benchmark experiment scenario of “hierarchical collaborative collection” in the field of multi-agent reinforcement learning, such as Figure 4As shown in (a) to (f) in the figure, the task goal is for multiple agents (white) to collaboratively collect multiple targets (red apples); each agent and target has a level (the number in the lower right corner), and the collection can only be completed when the sum of the levels of a single agent / multiple aggregated agents is higher than or equal to the target level; the simulation environment is set to the following three types: 11×11-6N-4F-S3, 15×15-8N-5F-S3, and 20×20-10N-6F-S3, where N is the number of agents, F is the number of targets, and S is the grid area of ​​local perception.

[0106] 2) Agent action settings: discrete action types, 6 action types (forward, backward, left, right, stop, collect); Agent generated interaction information settings: in the S×S grid area centered on itself, each agent and the target (p x ,p y ,l), where p x ,p y represents the two-dimensional grid coordinates, l represents the level; the reward is set as follows: when one or more agents cooperate to collect a target, the level of food (i.e., the target) will be rewarded to each agent weighted according to the level of each agent, and then the reward is normalized. If all targets have been collected, the sum of the rewards is one; Table 1 shows the comparison results of the agent interaction information aggregation method based on global state approximation in this application and the existing benchmark method (the average results obtained from 10,000 tests).

[0107] Table 1 Experimental comparison results

[0108]

[0109] 3) Results:

[0110] Table 1 shows the comparison results of the agent interaction information aggregation method based on global state approximation in this application and different benchmark methods in the task of "hierarchical collaborative collection". The agent interaction information aggregation method based on global state approximation in this application and the benchmark method are trained in three sets of scenarios: 11×11-6N-4F-S3 / 15×15-8N-5F-S3 / 20×20-10N-6F-S3. The aggregate representation generated by this application needs to be input into the strategy generation module to complete the agent selection and execution of actions. The strategy generation module used in this application is the benchmark QMIX. During the training process, the initial positions of the agent and the target are random. The number of training rounds is 200,000 rounds, and the maximum number of steps for each round of interaction with the environment is 25 steps. Among them, there are two comparison indicators, the indicator mean return: represents the average cumulative reward obtained by the multi-agent team; the indicator AEL: represents the time step required to complete the task; the existing benchmark methods used for comparison include SARNET, MAGIC, and MASIA. Among them, SARNET uses a memory-based attention network to consider past interaction experiences and infer the value of received communication messages; MAGIC uses a graph attention network to build a message processor to process communication messages; MASIA aggregates received messages into a compact representation that is highly correlated with local strategies; OURS represents the agent interaction information aggregation method based on global state approximation in this application.

[0111] 4) Conclusion:

[0112] It can be seen that the agent interaction information aggregation method based on global state approximation in the present application has significant improvements over the existing models in both the team's average cumulative reward mean return and the task completion time step; in the task setting of 20×20-10N-6F-S3, the agent interaction information aggregation method based on global state approximation in the present application has the most obvious advantages. Compared with the benchmark method, it obtains the largest average reward and can complete the task the fastest; with the increase of scene grid area, the number of agents, and the number of targets, the difficulty of the hierarchical collaborative collection task increases. Only the agent interaction information aggregation method based on global state approximation in the present application can complete the hierarchical collaborative collection task with a success rate of more than 95%, which reflects that the agent interaction information aggregation method based on global state approximation in the present application achieves better collaboration effects through effective interaction information aggregation.

[0113] Based on the same inventive concept, the embodiment of the present application also provides a system for aggregating agent interaction information based on global state approximation for implementing the above-mentioned method for aggregating agent interaction information based on global state approximation. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more embodiments of the system for aggregating agent interaction information based on global state approximation provided below can refer to the limitations of the method for aggregating agent interaction information based on global state approximation above, and will not be repeated here.

[0114] In an exemplary embodiment, Figure 5 As shown, a system for aggregating agent interaction information based on global state approximation is provided, comprising: a scene determination and initialization unit 1, a target data acquisition unit 2, a preprocessing unit 3 and an agent interaction information aggregation result prediction unit 4.

[0115] A scene determination and initialization unit 1 is used to determine the current hierarchical collaborative collection task scene and initialize the current hierarchical collaborative collection task scene; the hierarchical collaborative collection task scene includes multiple agents and multiple targets;

[0116] Target data acquisition unit 2, used to acquire target data; the target data includes the interaction information generated by the current agent itself in the initialized current hierarchical collaborative collection task scene and the interaction information set transmitted by the teammate agent; the teammate agent is the agent other than the current agent in the initialized hierarchical collaborative collection task scene;

[0117] A preprocessing unit 3, used for preprocessing the target data to obtain preprocessed target data;

[0118] The agent interaction information aggregation result prediction unit 4 is used to input the preprocessed target data into the agent interaction information aggregation model to aggregate the agent interaction information, and output the agent interaction information aggregation result prediction value; the agent interaction information aggregation model is obtained by training the neural network model with training samples; the training samples include the interaction information samples generated by the current agent itself, the interaction information set samples transmitted by the teammate agent and the agent interaction information aggregation result sample values; wherein, the agent interaction information aggregation model includes a graph convolutional network, a gated recurrent unit and an encoder connected in sequence.

[0119] As an optional implementation, the preprocessing unit 3 includes:

[0120] The feature matrix construction subunit 31 is used to construct a feature matrix based on the interaction information generated by the current agent itself and the set of interaction information transmitted by the teammate agents.

[0121] The adjacency matrix construction subunit 31 is used to input the interaction information generated by the current intelligent agent itself and the set of interaction information transmitted by the teammate intelligent agent into the attention mechanism network, output the attention weight of each intelligent agent to each interaction information, and construct the adjacency matrix according to the attention weight of each intelligent agent to each interaction information, and use the feature matrix and the adjacency matrix as the preprocessed target data.

[0122] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used for target data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for aggregating information of intelligent agents based on global state approximation is implemented.

[0123] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0124] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0125] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0127] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0128] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0129] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for aggregating agent interaction information based on global state approximation, It is characterized in that The agent interaction information aggregation method based on global state approximation includes: Determine a current hierarchical collaborative collection task scenario and initialize the current hierarchical collaborative collection task scenario; the hierarchical collaborative collection task scenario includes multiple agents and multiple targets; Acquire target data; the target data includes interaction information generated by the current agent itself in the initialized current hierarchical collaborative collection task scene and a set of interaction information transmitted by teammate agents; the teammate agents are agents other than the current agent in the initialized hierarchical collaborative collection task scene; Preprocessing the target data to obtain preprocessed target data; The preprocessed target data is input into the agent interaction information aggregation model to aggregate the agent interaction information, and the predicted value of the agent interaction information aggregation result is output; the agent interaction information aggregation model is obtained by training the neural network model with training samples; the training samples include the interaction information samples generated by the current agent itself, the interaction information set samples transmitted by the teammate agent, and the agent interaction information aggregation result sample value; Among them, the agent interaction information aggregation model includes a graph convolutional network, a gated recurrent unit and an encoder connected in sequence.

2. According to the method for aggregating agent interaction information based on global state approximation according to claim 1, It is characterized in that Initialize the current hierarchical collaborative collection task scenario, including: The grid size, number of agents, and number of targets of the current hierarchical collaborative collection task scenario are initialized, and the maximum number of steps for the agent to perform actions in the hierarchical collaborative collection task scenario is set to obtain the initialized hierarchical collaborative collection task scenario.

3. The agent interaction information aggregation method based on global state approximation according to claim 1, It is characterized in that Preprocessing the target data to obtain preprocessed target data specifically includes: Construct a feature matrix based on the interaction information generated by the current agent and the set of interaction information transmitted by teammate agents; The interaction information generated by the current agent itself and the set of interaction information transmitted by the teammate agent are input into the attention mechanism network, and the attention weight of each agent to each interaction information is output. An adjacency matrix is ​​constructed according to the attention weight of each agent to each interaction information, and the feature matrix and the adjacency matrix are used as preprocessed target data.

4. The agent interaction information aggregation method based on global state approximation according to claim 3, It is characterized in that The training process of the agent interaction information aggregation model specifically includes: Construct training samples based on the interaction information samples generated by the current agent and the interaction information set samples transmitted by teammate agents; Inputting the training sample into the graph convolutional network to obtain a graph representation at the current moment; Obtaining historical graph representations of the previous k moments, and inputting the historical graph representations of the previous k moments and the graph representation of the current moment into the gated recurrent unit to obtain a historical time series representation; Inputting the current graph representation and the historical time series representation into the encoder to obtain an aggregate representation; Inputting the aggregate representation into a decoder to obtain a decoded output of the aggregate representation; The global scene state sample value of the initialized hierarchical collaborative collection task scene is obtained, and the loss value is calculated based on the decoded output of the aggregate representation and the global scene state sample value, and the graph convolutional network parameters, gated recurrent unit parameters, encoder parameters, and decoder parameters are iteratively updated according to the loss value until the loss value is less than a preset threshold or the number of iterations reaches a maximum value, then the iterative update is stopped, and the trained graph convolutional network, gated recurrent unit, and encoder are used as an agent interaction information aggregation model.

5. According to claim 4, the method for aggregating agent interaction information based on global state approximation, It is characterized in that The loss value is calculated using the mean square error loss function, and the expression of the mean square error loss function is: in, represents the loss value; μ represents the encoder parameter; η represents the decoder parameter; Express expectation; t Represents the global state of the hierarchical collaborative collection scenario; represents the global aggregate representation; represents the decoded output of the global aggregate representation; s t Represents the global scene state of the hierarchical collaborative collection task scene after initialization; Represents the square of the L2 norm.

6. A system for aggregating information about agent interactions based on global state approximation, It is characterized in that The agent interaction information aggregation system based on global state approximation is based on the agent interaction information aggregation method based on global state approximation according to any one of claims 1 to 5, and the agent interaction information aggregation system based on global state approximation comprises: A scene determination and initialization unit, used to determine the current hierarchical collaborative collection task scene and initialize the current hierarchical collaborative collection task scene; the hierarchical collaborative collection task scene includes multiple agents and multiple targets; A target data acquisition unit, used to acquire target data; the target data includes interaction information generated by the current agent itself in the initialized current hierarchical collaborative collection task scene and a set of interaction information transmitted by teammate agents; the teammate agents are agents other than the current agent in the initialized hierarchical collaborative collection task scene; A preprocessing unit, used for preprocessing the target data to obtain preprocessed target data; An agent interaction information aggregation result prediction unit is used to input the preprocessed target data into the agent interaction information aggregation model to aggregate the agent interaction information and output the agent interaction information aggregation result prediction value; the agent interaction information aggregation model is obtained by training the neural network model with training samples; the training samples include the interaction information samples generated by the current agent itself, the interaction information set samples transmitted by the teammate agents and the agent interaction information aggregation result sample values; Among them, the agent interaction information aggregation model includes a graph convolutional network, a gated recurrent unit and an encoder connected in sequence.

7. The agent interaction information aggregation system based on global state approximation according to claim 6, It is characterized in that The pre-processing unit includes: The feature matrix construction subunit is used to construct a feature matrix based on the interaction information generated by the current agent itself and the set of interaction information transmitted by the teammate agents; The adjacency matrix construction subunit is used to input the interaction information generated by the current intelligent agent itself and the set of interaction information transmitted by the teammate intelligent agent into the attention mechanism network, output the attention weight of each intelligent agent to each interaction information, and construct the adjacency matrix according to the attention weight of each intelligent agent to each interaction information, and use the feature matrix and the adjacency matrix as the preprocessed target data.

8. A computer device, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the agent interaction information aggregation method based on global state approximation as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for aggregating agent interaction information based on global state approximation described in any one of claims 1 to 5 is implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method for aggregating agent interaction information based on global state approximation described in any one of claims 1 to 5 is implemented.

Images