Agent information interaction object determination method and system, electronic device, and medium

Abstract

This invention discloses a method, system, electronic device, and medium for determining information interaction objects between intelligent agents. It involves acquiring the self-observation information of the target intelligent agent and the position information of the current non-target intelligent agents in the current collaborative scenario; based on the self-observation information of the target intelligent agent and the position information of the current non-target intelligent agents, using an interaction preference network, predicting the interaction probability between the target intelligent agent and the non-target intelligent agents, and the non-interaction probability between the target intelligent agent and the current non-target intelligent agents; and determining the interaction objects of the target intelligent agent based on the interaction probability and the non-interaction probability. Specifically, KL divergence is used to measure the interaction preferences between agents, and the interaction preference network is trained using a training sample set determined by the interaction preference values. This invention improves the efficiency of information interaction between multiple intelligent agents.

Description

Technical Field

[0001] This invention relates to the field of intelligent agent information interaction, and in particular to a method, system, electronic device, and medium for determining intelligent agent information interaction objects. Background Technology

[0002] Multi-agent information interaction is crucial for collaboration among distributed multi-agent systems in the real world, such as autonomous vehicle planning, traffic light control, and multi-robot control. Especially in environments with incomplete observability, agents can only perceive and understand local scenes due to limited sensor range and sensor data noise. Information interaction between agents can provide them with a near-global perspective through information sharing, improving their understanding of the cluster's task execution and promoting collaborative behavior.

[0003] Existing multi-agent information interaction methods represent their own local observations by constructing communication neural networks, bidirectional coordination networks, and intent-sharing schemes, and then broadcast these representations to all other teammate agents. To achieve an adaptive interaction process, the TarMAC method introduces a signature-based attention mechanism, enabling agents to focus more on information relevant to themselves during information fusion. However, it remains a fully interactive mechanism, allowing agents to ignore irrelevant messages. Most multi-agent information interaction methods require information sharing among all agents, making it difficult for each agent to distinguish between valuable information that aids collaborative decision-making and redundant information that hinders it. This increases the difficulty of information filtering and fusion, hindering efficient collaboration and even impeding the learning process of collaborative strategies. Furthermore, in real-world applications, broadcast interaction requires a large amount of information transmission, is highly dependent on communication bandwidth, and is prone to information transmission blockage under weak communication conditions, causing information delays and loss, further complicating the learning of collaborative strategies.

[0004] In summary, current multi-agent interaction methods primarily employ observation information representation and broadcast transmission mechanisms for information sharing among agents. However, they fail to consider the varying interaction value of each object under different scenario states, resulting in a lack of targeted interaction, high bandwidth dependence on large amounts of information transmission, and significant communication overhead. Therefore, it is crucial to develop targeted information interaction processes that dynamically select valuable interaction objects for information sharing based on different collaborative states to improve the efficiency of agent interaction and collaboration. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, electronic device, and medium for determining information interaction objects between intelligent agents, so as to improve the efficiency of information interaction between multiple intelligent agents.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A method for determining information interaction objects of an intelligent agent includes:

[0008] The system acquires the self-observation information of the target intelligent agent and the current position information of the non-target intelligent agents in the collaborative scenario. The collaborative scenario includes multiple intelligent agents, multiple dynamic targets, and multiple static obstacles. The self-observation information includes the speed information of the target intelligent agent, the position information of the target intelligent agent, the position information of the static obstacles within the local perception range of the target intelligent agent, and the position information of the dynamic targets within the local perception range of the target intelligent agent.

[0009] Based on the target agent's own observation information and the current non-target agent's position information, an interaction preference network is used to predict the interaction probability between the target agent and the non-target agent, and the non-interaction probability between the target agent and the current non-target agent. The interaction preference network is determined by training an initial interaction preference network using a training sample set. The training sample set includes multiple training samples with interaction labels. Each training sample includes the target agent's own observation information and the position information of a non-target agent sample. The interaction label is 1 or 0; 1 indicates selecting the current non-target agent sample as the target agent's interaction target; 0 indicates not selecting the current non-target agent sample as the target agent's interaction target. The initial interaction preference network includes a first fully connected layer, a second fully connected layer, a third fully connected layer, and a softmax layer connected sequentially.

[0010] The interaction object of the target agent is determined based on the interaction probability and the non-interaction probability.

[0011] Optionally, the initial network for interaction preferences is trained using the training sample set, specifically including:

[0012] Obtain multiple training samples;

[0013] The training samples are labeled with interactive tags to construct a training sample set;

[0014] The training samples with interaction labels in the training sample set are input into the current interaction preference initialization network to determine the interaction probability between the target agent sample and the non-target agent sample and the non-interaction probability between the target agent sample and the non-target agent sample.

[0015] Calculate the interaction probability between the target agent sample and the non-target agent sample and the loss value of the sample label;

[0016] Update the parameters of the initial network for the current interaction preference based on the loss value;

[0017] Determine if the current training round number is greater than the maximum training round number. If yes, end the training. If no, return to the step of "inputting a batch of training samples with interaction labels from the training sample set into the current interaction preference initial network to determine the interaction probability between the target agent sample and the non-target agent sample and the non-interaction probability between the target agent sample and the non-target agent sample".

[0018] Optionally, interactive labels are annotated on the training samples to construct a training sample set, specifically including:

[0019] The training samples are sequentially input into the interaction preference initialization network to determine the candidate interaction objects of the target agent sample;

[0020] For any target agent sample:

[0021] The system receives the first interaction information sent by the candidate interaction object and determines the second interaction information to be sent to other intelligent agents that selected the target intelligent agent sample as a candidate interaction object based on the target intelligent agent sample's own observation information.

[0022] Based on the target agent's own observation information and the first interaction information, the action selected at the current moment is determined and executed to obtain the collaborative scene after the action is executed; the action is forward, backward, left, right or remain stationary;

[0023] Calculate the actions selected by the target agent sample and the actions selected by the candidate interaction object, and determine the softmax distribution and marginal distribution of the actions;

[0024] Based on the softmax distribution and the marginal distribution, the interaction preference values ​​between the target agent sample and the candidate interaction object are determined; the interaction preference values ​​are the KL divergence between the softmax distribution and the marginal distribution.

[0025] Based on the interaction preference value and the preset preference threshold, the interaction labels of the target agent sample and the candidate interaction object are determined to obtain the training sample set.

[0026] Optionally, based on the interaction preference value and a preset preference threshold, the interaction labels of the target agent sample and the candidate interaction object are determined to obtain a training sample set, specifically including:

[0027] Determine whether the interaction preference value is greater than the preset preference threshold;

[0028] If so, then the interaction label between the target agent sample and the candidate interaction object is determined to be 1;

[0029] If not, then the interaction label between the target agent sample and the candidate interaction object is determined to be 0.

[0030] Optionally, determining the preset preference threshold specifically includes:

[0031] Sort the target agent samples and the interaction preference values ​​of each candidate interaction object in ascending order;

[0032] The interaction preference value ranked in the preset quantile is used as the preset preference threshold.

[0033] Optionally, the interaction object of the target agent is determined based on the interaction probability and the non-interaction probability, specifically including:

[0034] Determine whether the probability of interaction is greater than the probability of no interaction;

[0035] If so, then the current non-target agent is determined to be the interaction object of the target agent;

[0036] If not, then it is determined that the current non-target agent is not the interaction object of the target agent.

[0037] A system for determining intelligent agent information interaction objects, comprising:

[0038] The information acquisition module is used to acquire the self-observation information of the target intelligent agent and the current position information of the non-target intelligent agent in the collaborative scenario; the collaborative scenario is set with multiple intelligent agents, multiple dynamic targets and multiple static obstacles.

[0039] A probability prediction module is used to predict the interaction probability between the target agent and the non-target agent and the non-interaction probability between the target agent and the current non-target agent based on the target agent's own observation information and the current non-target agent's position information, using an interaction preference network. The interaction preference network is determined by training an initial interaction preference network using a training sample set. The training sample set includes multiple training samples with interaction labels. Each training sample includes the target agent's own observation information and the position information of a non-target agent sample. The interaction label is 1 or 0; 1 indicates selecting the current non-target agent sample as the interaction object of the target agent sample; 0 indicates not selecting the current non-target agent sample as the interaction object of the target agent sample. The initial interaction preference network includes a first fully connected layer, a second fully connected layer, a third fully connected layer, and a softmax layer connected sequentially.

[0040] The interaction object determination module is used to determine the interaction object of the target agent based on the interaction probability and the non-interaction probability.

[0041] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor runs the computer program to cause the electronic device to perform the above-described method for determining intelligent agent information interaction objects.

[0042] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for determining intelligent agent information interaction objects.

[0043] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0044] The present invention discloses a method, system, electronic device, and medium for determining intelligent agent information interaction objects. This involves acquiring the self-observation information of a target intelligent agent and the position information of a non-target intelligent agent in the current collaborative scenario. Based on the self-observation information of the target intelligent agent and the position information of the non-target intelligent agent, an interaction preference network is used to predict the interaction probability between the target intelligent agent and the non-target intelligent agent, and the non-interaction probability between the target intelligent agent and the non-interaction probability. The interaction object of the target intelligent agent is then determined based on the interaction probability and the non-interaction probability. Specifically, KL divergence is used to measure the interaction preference between agents, and the interaction preference network is trained using a training sample set determined by the interaction preference values. In multi-agent collaboration, the introduction of interaction preferences allows agents to adaptively select different interaction objects for information sharing under different scenario states, forming a targeted information interaction process. Information sharing enhances the agent's perception and understanding of the global scenario and its teammates, alleviating the problem of incomplete scene observability and thus improving the efficiency of information interaction between multiple agents. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 Flowchart of the method for determining intelligent agent information interaction objects provided by the present invention;

[0047] Figure 2 This is a flowchart of the initial network training process for interaction preferences in this invention;

[0048] Figure 3 A framework diagram of the multi-agent information interaction object determination method provided by the present invention;

[0049] Figure 4 This is a diagram illustrating the task execution effect in the "cooperative encirclement" scenario of a simulation experiment. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] The purpose of this invention is to provide a method, system, electronic device, and medium for determining information interaction objects between intelligent agents, so as to improve the efficiency of information interaction between multiple intelligent agents.

[0052] By collecting sample data through interaction with the "cooperative encirclement" scenario, the sample data is then labeled according to the defined interaction preference calculation formula. A loss function for the interaction preference network is constructed, and a batch of sample data is randomly selected. The loss value is calculated according to the loss function, and the initial interaction preference network is trained. This allows the agent to filter valuable interaction objects based on the interaction preference network during the interaction process. It can select different interaction objects for information sharing based on the observation situation at different times, avoiding the interaction redundancy caused by the broadcast interaction mechanism and saving the interaction overhead caused by the large amount of information transmission in the broadcast interaction. Finally, by integrating the interaction information provided by high-value interaction objects, the agent can obtain more complete environmental information, alleviate the problem of incomplete observation of the cooperative encirclement scenario, and cooperate more efficiently to complete the encirclement task.

[0053] Compared with other multi-agent information interaction methods, especially broadcast interaction methods, the method described in this invention, based on the introduction of an interaction preference network, actively selects teammates who can provide interaction value as interaction objects according to different scenario states during the interaction process. This facilitates the sharing of local observation information representations and improves interaction efficiency through targeted interaction methods, resulting in more concise interaction information and reducing the complexity of collaborative strategy inputs. By selecting interaction objects, the model can reduce system communication redundancy and overhead, avoid decision delays caused by large amounts of message transmission, and make it easier to deploy in the real world.

[0054] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] Example 1

[0056] like Figure 1 As shown, the method for determining the intelligent agent information interaction object provided by the present invention includes:

[0057] Step 101: Obtain the self-observation information of the target intelligent agent and the current position information of the non-target intelligent agents in the collaborative scenario; the collaborative scenario includes multiple intelligent agents, multiple dynamic targets, and multiple static obstacles. The self-observation information includes the velocity information of the target intelligent agent, the position information of the target intelligent agent, the position information of the static obstacles within the local perception range of the target intelligent agent, and the position information of the dynamic targets within the local perception range of the target intelligent agent.

[0058] Step 102: Based on the target agent's own observation information and the current non-target agent's position information, use an interaction preference network to predict the interaction probability between the target agent and the non-target agent and the non-interaction probability between the target agent and the current non-target agent; wherein, the interaction preference network is determined by training an initial interaction preference network using a training sample set; the training sample set includes multiple training samples with interaction labels; the training samples include the target agent's own observation information and the position information of a non-target agent sample; the interaction label is 1 or 0; 1 indicates that the current non-target agent sample is selected as the interaction object of the target agent sample; 0 indicates that the current non-target agent sample is not selected as the interaction object of the target agent sample; the initial interaction preference network includes a first fully connected layer, a second fully connected layer, a third fully connected layer, and a softmax layer connected in sequence.

[0059] Step 103: Determine the interaction object of the target agent based on the interaction probability and the non-interaction probability.

[0060] In practical applications, such as Figure 2 As shown, the initial network for interaction preferences is trained using the training sample set, specifically including:

[0061] Step 1: Initialize the "cooperative encirclement" task scenario and the interaction preference network parameters. The specific process is as follows:

[0062] (1-1) Initialize the “cooperative scenario”: Initialize the scenario with N agents, L dynamic targets, and K static obstacles. Set the maximum number of training rounds to NUM_EPISODES and the maximum number of execution steps in one round to NUM_STEPS. Then, initialize the positions and velocities of N agents, L dynamic targets, and K static obstacles.

[0063] (1-2) Initialize the initial network parameters for interaction preferences: The specific network structure for the initial interaction preference network is as follows: the input includes the local observation information (self-observation information) of the target agent sample i. i Location information u of non-target agent sample j (teammate) j The process involves three fully connected layers (the last layer outputs two units) and a softmax layer (outputting two units). The two output units represent the probability that target agent sample i does not choose to interact with teammate j at the current time, and the probability that target agent sample i chooses to interact with teammate j.

[0064] Step 2: The agent interacts with the scene and stores the generated interaction samples in the experience pool. The specific process is as follows:

[0065] (2-1) Iterate through the number of steps NUM_STEPS that the agent team performs in the scene.

[0066] (2-2) For each step, iterate through the number of agents N in the team.

[0067] (2-3) For each target agent sample i, such as Figure 3 As shown, the observation information o of the target intelligent agent sample i itself is... i The location information of each non-target agent sample (all teammates) u j , in sequence combine {(o i ,u1),(o i ,u j ),…,(o i ,u N-1 The input is fed into the interaction preference initialization network, which outputs the normalized interaction probability and non-interaction probability between the target agent sample i and each non-target agent sample. If the interaction probability is greater than the non-interaction probability, then the target agent sample i selects to interact with the non-target agent sample j at the current time and obtains the number of the interaction object selected at the current time.

[0068] (2-4) Receive the first interaction information sent by the candidate interaction object, and determine the second interaction information to be sent to other intelligent agents who selected the target intelligent agent sample as a candidate interaction object based on the self-observation information of the target intelligent agent sample.

[0069] In practical applications, for each target agent sample i, the system receives interaction information sent by the candidate interaction objects selected through the interaction preference initialization network at the current time, and also receives the target agent sample i's own observation information o. i The input is fed into a pre-trained interaction information encoding network and encoded into interaction information m. i , and send it to other agents that have selected the target agent sample i as a candidate interaction object.

[0070] The input to the pre-trained interaction information encoding network is the agent's current self-observation information plus the interaction information received by the agent from its teammates in the previous time step.

[0071] The specific structure of the pre-trained interactive information encoding network consists of two fully connected layers.

[0072] The output of the pre-trained interactive information encoding network consists of 32 units, representing the encoded interactive information to be transmitted.

[0073] Using the classic multi-agent algorithm MADDPG (which includes a baseline action network, value network, and network update method) and a broadcast full-interaction approach (each agent sends encoded interaction information to all teammates), we pre-trained the interaction information encoding network in the same task scenario.

[0074] (2-5) Based on the target agent's own observation information and the first interaction information, determine the action selected at the current moment and execute it to obtain the collaborative scene after executing the action. The action is forward, backward, left, right, or remaining stationary.

[0075] In practical applications, for each target agent sample i, the observation information o of the target agent sample i itself is... i The received first interaction information is fused with the input of the pre-trained action network to output the action 'a' selected at the current time step. i And perform the action.

[0076] The input to the pre-trained action network consists of the agent's current self-observation information and the received interaction information.

[0077] The pre-trained action network consists of two fully connected layers.

[0078] The output of the pre-trained action network consists of 5 units, representing the action dimension, specifically the five actions of forward, backward, left, right, and hold still.

[0079] Using the classic multi-agent algorithm MADDPG with a broadcast full-interaction approach (each agent sends coded interaction information to all teammates), the action network is pre-trained in the same task scenario.

[0080] (2-6) Store the interaction sample (the agent's own observation information, the selected interaction object number, the selected interaction object location information, the action information executed, the new scene state obtained by executing the action, and the reward information obtained) into the experience pool.

[0081] Step 3: Label the sample data according to the defined interaction preference calculation formula. The specific process is as follows:

[0082] (3-1) Calculate the actions selected by the target agent sample and the actions selected by the candidate interaction object, and determine the softmax distribution and marginal distribution of the actions.

[0083] In practical applications, for the target agent sample i and each selected candidate interaction object, the softmax distribution of the action selected by the target agent sample i is calculated. This distribution represents the influence of agent i's interaction behavior on itself when selecting an action, where t represents the time t. This represents the action chosen by the target agent sample i at time t. Let represent the action chosen by agent j at time t, o represent the joint observation of all agents, and m represent the joint interaction information of all agents. Indicates the value of an action. This represents the counterfactual action of target agent sample i, that is, the action that target agent sample i did not choose to execute in this step. This represents the joint action that does not include the action of the target agent sample i. λ represents the temperature coefficient.

[0084] For the target agent sample i and each selected candidate interaction object, calculate the marginal distribution. This distribution indicates that when target agent sample i selects an action, it does not consider the influence of agent j's interaction behavior on itself, where t represents the time t. Let $\mathbf{i}$ represent the action chosen by target agent sample $i$ at time $t$, $\mathbf{i}$ represent the joint observations of all agents, and $\mathbf{i}$ represent the joint interaction information of all agents. This represents a joint action that does not include the actions of target agent sample i and agent j. This represents the counterfactual action of target agent sample i, that is, the action that target agent sample i did not choose to execute in this step. This represents the counterfactual action of agent j, which is the action that agent j did not choose to perform in this sample step.

[0085] (3-2) Determine the interaction preference values ​​of the target agent sample and the candidate interaction object based on the softmax distribution and the marginal distribution; the interaction preference values ​​are the KL divergence of the softmax distribution and the marginal distribution.

[0086] In practical applications, for a target agent sample i and each selected candidate interaction object, the interaction preference between the target agent sample i and the interaction object j is calculated, which is the KL divergence between the two distributions. If the KL divergence between two distributions is small, it indicates that the difference between the two distributions is small. This means that whether agent j interacts with target agent i or not, the impact on target agent i's own behavioral strategy is minimal. In this case, agent j has little interaction value with target agent i under the current circumstances. Therefore, target agent i has a low interaction preference for agent j. Conversely, if the KL divergence between the two distributions is large, it indicates that interaction between agent j and target agent i will have a significant impact on target agent i's behavioral strategy. In this case, agent j has significant interaction value with target agent i. Therefore, target agent i has a high interaction preference for agent j.

[0087] Based on the interaction preference value and the preset preference threshold, the interaction labels of the target agent sample and the candidate interaction object are determined to obtain the training sample set. Specifically, see (3-3) and (3-4).

[0088] (3-3) Collect the interaction preference values ​​calculated for all samples, sort them in ascending order, and obtain the interaction preference values ​​ranked at the 80th percentile as the preset preference threshold.

[0089] (3-4) Compare all interaction preference values ​​with the preset preference threshold. If the value is greater than the preset preference threshold, remove the interaction label. Setting it to 1 indicates that it is selected as the interaction object; otherwise, the interaction label will be removed. Setting it to 0 means that it is not selected as an interaction object.

[0090] Step 4: Construct the loss function for training the interaction preference network. The specific process is as follows:

[0091] (4-1) The loss function of the interaction preference network is defined as follows: in This indicates the need to update the interaction preference network parameter φ. c loss function, Represents variable o i ,u j Expected value This represents the label value; log indicates that the operation is performed by taking the logarithm. The parameter is represented by φ. c Interaction preference network, O represents the probability of interacting with teammate j predicted using an interaction preference network. i u represents the local observation information of the target agent sample i. jThis indicates the location information of the candidate interactive object.

[0092] Step 5: Randomly select a batch of sample data and input it into the interaction preference network. Combine the network's predicted interaction probabilities with the sample labels, input them into the loss function, calculate the loss value, and update the interaction preference network. The specific process is as follows:

[0093] (5-1) Randomly select a batch of sample data from the experience pool. The data format is as follows:

[0094]

[0095] (5-2) From a batch of sample data, the agent's own observation information o i Location information of candidate interaction objects u j The input is fed into the interaction preference network to obtain the network's predicted interaction probabilities.

[0096] (5-3) The interaction probability predicted by the interaction preference network in a batch of sample data. and sample labels Input defined loss function In the process, the loss value is obtained, and the interaction preference network is updated in reverse to complete one round of training.

[0097] Step 6: If the number of training rounds exceeds the set maximum number of training rounds, end the training; otherwise, repeat steps 2-5. The specific process is as follows:

[0098] (6-1) Determine if the current training round number is greater than the maximum training round number NUM_EPISODES. If it is greater than the maximum training round number, end the training.

[0099] (6-2) If the current training round number is less than or equal to the maximum training round number, repeat steps 2-5 to update the interaction preference network.

[0100] As an optional implementation, step 103 specifically includes:

[0101] Determine whether the probability of interaction is greater than the probability of no interaction.

[0102] If so, then the current non-target agent is determined to be the interaction object of the target agent.

[0103] If not, then it is determined that the current non-target agent is not the interaction object of the target agent.

[0104] Simulation experiments were conducted to illustrate the method of this invention:

[0105] The simulation environment is a benchmark experimental scenario in the field of multi-agent reinforcement learning, namely "cooperative encirclement" (source: https: / / github.com / openai / multiagent-particle-envs). The task objective is: [e.g., ...] Figure 4 As shown, multiple agents (red) surround and capture multiple enemy agents (green) without colliding. The simulation environment is set as follows: a 2D scene size of 20 units * 20 units, an agent size of 1 unit, an enemy agent size of 1 unit, and three agent number settings: 3 vs. 1, 4 vs. 2, and 5 vs. 2. Agent actions are set as follows: discrete action type, 5 action categories (forward, backward, left, right, stop). Agent observation... Settings: All information of the agent itself (including velocity and position coordinates), partial information of teammates (position information), partial information of enemy agents (position information), and information of static entities in the scene (position information of static obstacles); the actions of enemy agents are randomly generated, and automatic obstacle avoidance is enabled. The reward for our cooperative agent team is the sum of the negative distances between all agents and all enemy agents. Table 1 shows the comparison results between the method of this invention and existing benchmark methods (average results obtained from 10,000 tests).

[0106] Table 1. Statistical table of experimental results of the method of the present invention and existing benchmark methods.

[0107]

[0108] Table 1 shows the comparison results between the method of this invention and different benchmark methods in the task of "cooperative encirclement". The method of this invention and the benchmark method were trained in three settings with the number of agents: 3 vs. 1, 4 vs. 2, and 5 vs. 2. During the training process, the initial positions of the agents and the target point were randomized, the number of training rounds was 100,000, and the maximum number of steps to interact with the environment in each round was 50.

[0109] Two comparison metrics were used: S% represents the task success rate; R represents the final reward value obtained during training. Existing benchmark methods used for comparison included MADDPG, IC3Net, TarMaC, and I2C. MADDPG does not involve direct interaction between agents; IC3Net uses a gating mechanism to determine whether to send interaction information to other agents; and TarMaC uses a signature-based attention mechanism, enabling agents to focus more on information more relevant to themselves during interaction information fusion, but it is still a fully interactive mechanism, allowing agents to ignore irrelevant messages. OURS represents the method used in this invention.

[0110] It can be seen that the method of this invention significantly improves upon existing models in both success rate (S%) and reward value (R). In a task setting with 3 agents versus 1 agent, the method of this invention achieves the highest task completion rate and the largest training reward value. As the number of agents and enemy agents increases, the difficulty of the cooperative encirclement task rises. Only the method of this invention can complete the cooperative encirclement task with a success rate approaching 90%, demonstrating that the method of this invention achieves better cooperative results through efficient information interaction between agents.

[0111] This invention proposes a method, system, electronic device, and medium for determining information interaction objects among multiple agents. In the process of multi-agent collaboration, by introducing interaction preferences, agents can adaptively select different interaction objects for information sharing under different scenario states, forming a targeted information interaction process. Through information sharing, agents can improve their perception and understanding of the global scenario and their teammates, alleviating the problem of incomplete observability of the scenario, thereby completing the collaborative capture task more efficiently.

[0112] The basic features of this invention are as follows: First, an interaction preference network is introduced in the information interaction process, which enables the agent to autonomously select teammates with interaction value to interact with in different scenario states, forming a targeted interaction mechanism; Second, in the process of defining interaction preferences, KL divergence is used to measure the interaction preferences between agents, guiding the agent to learn to select teammates who have a greater impact on its own behavior strategy as the current high-value interaction object.

[0113] Example 2

[0114] In order to execute the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, an intelligent agent information interaction object determination system is provided below, including:

[0115] The information acquisition module is used to acquire the self-observation information of the target intelligent agent and the current position information of non-target intelligent agents in the collaborative scenario; the collaborative scenario includes multiple intelligent agents, multiple dynamic targets, and multiple static obstacles. The self-observation information includes the velocity information of the target intelligent agent, the position information of the target intelligent agent, the position information of static obstacles within the local perception range of the target intelligent agent, and the position information of dynamic targets within the local perception range of the target intelligent agent.

[0116] A probability prediction module is used to predict the interaction probability between the target agent and the non-target agent and the non-interaction probability between the target agent and the current non-target agent based on the target agent's own observation information and the current non-target agent's position information, using an interaction preference network. The interaction preference network is determined by training an initial interaction preference network using a training sample set. The training sample set includes multiple training samples with interaction labels. Each training sample includes the target agent's own observation information and the position information of a non-target agent sample. The interaction label is 1 or 0; 1 indicates selecting the current non-target agent sample as the target agent's interaction object; 0 indicates not selecting the current non-target agent sample as the target agent's interaction object. The initial interaction preference network includes a first fully connected layer, a second fully connected layer, a third fully connected layer, and a softmax layer connected sequentially.

[0117] The interaction object determination module is used to determine the interaction object of the target agent based on the interaction probability and the non-interaction probability.

[0118] Example 3

[0119] The present invention provides an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the intelligent agent information interaction object determination method of Embodiment 1.

[0120] Example 4

[0121] This invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the intelligent agent information interaction object determination method of Embodiment 1.

[0122] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0123] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for determining information interaction objects in an intelligent agent, characterized in that, include: Acquire the self-observation information of the target intelligent agent and the current position information of the non-target intelligent agents in the collaborative scenario; The collaborative scenario includes multiple intelligent agents, multiple dynamic targets, and multiple static obstacles. The self-observation information includes the velocity information of the target intelligent agent, the position information of the target intelligent agent, the position information of static obstacles within the local perception range of the target intelligent agent, and the position information of dynamic targets within the local perception range of the target intelligent agent. Based on the target agent's own observation information and the current non-target agent's position information, an interaction preference network is used to predict the interaction probability between the target agent and the non-target agent, and the non-interaction probability between the target agent and the current non-target agent. The interaction preference network is determined by training an initial interaction preference network using a training sample set. The training sample set includes multiple training samples with interaction labels. Each training sample includes the target agent's own observation information and the position information of a non-target agent sample. The interaction label is 1 or 0; 1 indicates selecting the current non-target agent sample as the target agent's interaction target; 0 indicates not selecting the current non-target agent sample as the target agent's interaction target. The initial interaction preference network includes a first fully connected layer, a second fully connected layer, a third fully connected layer, and a softmax layer connected sequentially. The interaction object of the target agent is determined based on the interaction probability and the non-interaction probability.

2. The method for determining the intelligent agent information interaction object according to claim 1, characterized in that, The initial network for interaction preferences is trained using a training sample set, specifically including: Obtain multiple training samples; The training samples are labeled with interactive tags to construct a training sample set; The training samples with interaction labels in the training sample set are input into the current interaction preference initialization network to determine the interaction probability between the target agent sample and the non-target agent sample and the non-interaction probability between the target agent sample and the non-target agent sample. Calculate the interaction probability between the target agent sample and the non-target agent sample and the loss value of the sample label; Update the parameters of the initial network for the current interaction preference based on the loss value; Determine if the current training round number is greater than the maximum training round number. If yes, end the training. If no, return to the step of "inputting the training samples with interaction labels in the training sample set into the current interaction preference initial network to determine the interaction probability between the target agent sample and the non-target agent sample and the non-interaction probability between the target agent sample and the non-target agent sample".

3. The method for determining the intelligent agent information interaction object according to claim 1, characterized in that, The training samples are labeled with interactive tags to construct a training sample set, specifically including: The training samples are sequentially input into the interaction preference initialization network to determine the candidate interaction objects of the target agent sample; For any target agent sample: The system receives the first interaction information sent by the candidate interaction object and determines the second interaction information to be sent to other intelligent agents that selected the target intelligent agent sample as a candidate interaction object based on the target intelligent agent sample's own observation information. Based on the target agent's own observation information and the first interaction information, the action selected at the current moment is determined and executed to obtain the collaborative scene after the action is executed; the action is forward, backward, left, right or remain stationary; Calculate the actions selected by the target agent sample and the actions selected by the candidate interaction object, and determine the softmax distribution and marginal distribution of the actions; Based on the softmax distribution and the marginal distribution, the interaction preference values ​​between the target agent sample and the candidate interaction object are determined; the interaction preference values ​​are the KL divergence between the softmax distribution and the marginal distribution. Based on the interaction preference value and the preset preference threshold, the interaction labels of the target agent sample and the candidate interaction object are determined to obtain the training sample set.

4. The method for determining the information interaction object of an intelligent agent according to claim 3, characterized in that, Based on the interaction preference value and the preset preference threshold, the interaction labels of the target agent sample and the candidate interaction object are determined to obtain a training sample set, specifically including: Determine whether the interaction preference value is greater than the preset preference threshold; If so, then the interaction label between the target agent sample and the candidate interaction object is determined to be 1; If not, then the interaction label between the target agent sample and the candidate interaction object is determined to be 0.

5. The method for determining the intelligent agent information interaction object according to claim 3, characterized in that, The determination of the preset preference threshold specifically includes: Sort the target agent samples and the interaction preference values ​​of each candidate interaction object in ascending order; The interaction preference value ranked in the preset quantile is used as the preset preference threshold.

6. The method for determining the information interaction object of an intelligent agent according to claim 1, characterized in that, Based on the interaction probability and the non-interaction probability, the interaction object of the target agent is determined, specifically including: Determine whether the probability of interaction is greater than the probability of no interaction; If so, then the current non-target agent is determined to be the interaction object of the target agent; If not, then it is determined that the current non-target agent is not the interaction object of the target agent.

7. A system for determining intelligent agent information interaction objects, characterized in that, include: The information acquisition module is used to acquire the target intelligent agent's own observation information and the current position information of non-target intelligent agents in the collaborative scenario; The collaborative scenario includes multiple intelligent agents, multiple dynamic targets, and multiple static obstacles. A probability prediction module is used to predict the interaction probability between the target agent and the non-target agent and the non-interaction probability between the target agent and the current non-target agent based on the target agent's own observation information and the current non-target agent's position information, using an interaction preference network. The interaction preference network is determined by training an initial interaction preference network using a training sample set. The training sample set includes multiple training samples with interaction labels. Each training sample includes the target agent's own observation information and the position information of a non-target agent sample. The interaction label is 1 or 0; 1 indicates selecting the current non-target agent sample as the interaction object of the target agent sample; 0 indicates not selecting the current non-target agent sample as the interaction object of the target agent sample. The initial interaction preference network includes a first fully connected layer, a second fully connected layer, a third fully connected layer, and a softmax layer connected sequentially. The interaction object determination module is used to determine the interaction object of the target agent based on the interaction probability and the non-interaction probability.

8. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to cause the electronic device to perform the intelligent agent information interaction object determination method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the intelligent agent information interaction object determination method according to any one of claims 1-6.

Images