Multi-agent decision-making method based on active reasoning

Abstract

The invention provides a multi-agent decision-making method based on active reasoning. The method comprises the steps that a teammate active reasoning network is constructed, the teammate active reasoning network models a teammate active reasoning process from the three aspects of perception, belief and action by means of a main agent, and observation perception representation of the teammate agent to the environment, cognitive belief representation of the teammate agent to the environment and prediction action of the teammate agent to the next time step are obtained; the main agent screens the sensing representations of the modeled teammate agents, unreliable sensing representations and corresponding teammate agents are screened out, and the reserved teammate agents serve as collaboration objects; and the main agent fuses the belief representations of the selected collaborative objects, and makes a decision according to the fusion result and the historical trajectory representation of the main agent. According to the method, the modeling information is selected by modeling the active reasoning process of the teammate agents and combining a dual screening mechanism, and the task execution efficiency and success rate of the multi-agent system in a dynamic complex environment are remarkably improved by reasonably modeling teammate behaviors and screening high-credibility collaborators.

Description

Technical Field

[0001] This invention relates to the field of multi-agent reinforcement learning technology, and in particular to a multi-agent decision-making method based on active reasoning. Background Technology

[0002] Multi-agent reinforcement learning (MARL), as an extension of deep reinforcement learning to multi-agent scenarios, studies how multiple agents can learn policies that achieve global goals through interaction with the environment and cooperation or competition with each other. Compared with single-agent reinforcement learning, MARL faces more complex challenges. In single-agent scenarios, agents only need to optimize their policies to maximize cumulative rewards, the dynamics of the environment are relatively fixed, and the learning process of agents can be modeled using Markov Decision Processes (MDPs). However, in multi-agent scenarios, the behavior of agents not only affects their own environment but also indirectly influences the observations, policies, and rewards of other agents. This interaction makes the environment complex and highly non-stationary.

[0003] In multi-agent systems, the non-stationarity of the environment manifests in several ways. First, the decisions and actions of each agent directly affect the observation space of other agents, causing the state transition probability of the environment to be determined not only by the policy of a single agent, but by the combined effect of the policies of multiple agents. This policy dependency causes the state distribution of the environment to change dynamically over time, resulting in a high degree of non-stationarity, which contrasts sharply with the assumption of relatively fixed environmental dynamics in traditional single-agent reinforcement learning. Second, the non-stationarity in multi-agent systems is also reflected in the dynamics of policies among agents. In multi-agent reinforcement learning, each agent independently learns and optimizes its own policy, while the policies of other agents are constantly changing. This dynamic interaction of policies makes the global dynamics of the environment complex and unpredictable, making it difficult to directly apply the fixed environment model-based methods of traditional single-agent reinforcement learning to such scenarios. Agents in this dynamic environment need to constantly adapt to the behavioral changes of other agents, which places higher demands on the convergence and stability of the learning algorithm.

[0004] One existing method for addressing nonstationarity in multi-agent reinforcement learning involves employing a Centralized Training with Decentralized Execution (CTDE) paradigm. CTDE mitigates nonstationarity issues caused by partial observability and policy dynamics during training by incorporating global information into the training phase. Within the CTDE framework, agents can access global state information during the centralized training phase, enabling them to learn cooperative policies more effectively and reducing conflicts arising from information gaps or policy changes.

[0005] The drawbacks of one of the existing methods for solving the non-stationarity problem in multi-agent reinforcement learning include: in the actual execution phase, the agents still rely only on local observations to make decisions and cannot obtain global information or the real-time policies of other agents. This makes the non-stationarity problem still exist in the execution phase, which limits the performance of multi-agent systems in practical applications. Summary of the Invention

[0006] This invention provides a multi-agent decision-making method based on active reasoning, thereby achieving efficient and reliable policy coordination in multi-agent systems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] A multi-agent decision-making method based on active reasoning includes:

[0009] A teammate active reasoning network is constructed. The teammate active reasoning network uses the agent to model the teammate active reasoning process from three levels: perception, belief and action. It obtains the teammate agent's observation and perception representation of the environment, the cognitive belief representation of the environment and the predicted action of the next time step.

[0010] The main intelligent agent filters the perceptual representations of the modeled teammate intelligent agents, filtering out unreliable perceptual representations and their corresponding teammate intelligent agents, and retaining the teammate intelligent agents as the cooperation objects.

[0011] The main intelligent agent fuses the belief representations of the selected collaborating objects and makes decisions based on the fusion result and its own historical trajectory representation.

[0012] Preferably, the construction of the teammate active reasoning network utilizes the agent to model the teammate's active reasoning process from three levels: perception, belief, and action, to obtain the teammate agent's observational perception representation of the environment, cognitive belief representation of the environment, and predicted action for the next time step, including:

[0013] A teammate active reasoning network is constructed, comprising a perception modeling network, a belief modeling network, and an action prediction network. The perception modeling network uses a coordinate transformation strategy to construct a relative observation space, setting the position of each teammate agent as the origin, and recalculating the relative position information of other agents relative to that teammate agent. The belief modeling network takes trajectory representation and teammate index as input and generates a belief distribution through a learnable belief encoder. The action prediction network takes belief representation and historical perception information as input, integrates the two types of key information through feature concatenation, and uses a multilayer perceptron (MLP) network for nonlinear transformation to generate the action distribution of teammates.

[0014] The agent modeled the teammate's active reasoning process from three levels: perception, belief, and action. This yielded the teammate's observational perception representation of the environment, its cognitive belief representation of the environment, and its predicted action for the next time step. The agent processed the teammate's historical trajectory data through a gated recurrent unit (GRU) network, extracted temporal features, and fused them with current observation information to generate a teammate's perception representation containing spatiotemporal context. The agent modeled each teammate's belief representation based on its own historical trajectory information and teammate ID index. Based on the modeling results of the teammate's perception and belief representations, the agent modeled the teammate's action at the current moment.

[0015] Preferably, the primary intelligent agent filters the perceptual representations of the modeled teammate intelligent agents, filtering out unreliable perceptual representations and their corresponding teammate intelligent agents, and retaining the teammate intelligent agents as cooperation objects, including:

[0016] A dual-filtering network is constructed, comprising an accuracy filtering module and a relevance filtering module. The accuracy filtering module is based on an accuracy scoring mechanism using a mapping function f:R. h →R maps the teammate agent's perceptual representation to a scalar value, which represents the teammate agent's accuracy score. At time step t, N×N perceptual representations are processed simultaneously to construct an evaluation matrix. Where N represents the number of agents. This represents the perception representation of agent i towards agent j. The accuracy score is specifically defined as:

[0017]

[0018] Where f(·) represents a two-layer multilayer perceptron network used to learn the mapping from perceptual representation to accuracy score, and c ij The higher the value, the more accurate the perception representation of agent i is to agent j. The top_k teammate agents with the highest accuracy scores are selected as the cooperation objects.

[0019] Preferably, the main intelligent agent fuses the belief representations of the selected collaborative objects, including:

[0020] The correlation filtering module targets each teammate agent as a collaboration partner, and retrieves the actual perception history of teammate agent i. As a query, the historical perception representations of the selected k teammate agents are used. As the key, the beliefs of these k teammate agents are represented. As the value, the relevance between each teammate agent's belief representation and the query of teammate agent i is calculated through an attention mechanism, and the attention score α is used. ij The calculation formula is as follows:

[0021]

[0022] Among them, W Q and W K These are respectively applied to the query. s and keys The learnable weight matrix, d k This represents the dimension of the key vector. Attention score α i，j This indicates the degree of correlation between agent i and teammate j. The higher the value, the more important teammate j is to agent i's current decision.

[0023] The belief representations of each collaborating object are weighted and fused. The weighted fusion result based on the attention mechanism is represented as follows:

[0024]

[0025] in Let represent the belief feature vector obtained by weighted fusion of the belief representations of the various teammate agents of the active agent i. This represents the belief of teammate agent j.

[0026] Preferably, the decision-making based on the fusion result and its own historical trajectory representation includes:

[0027] The main intelligent agent i will The real perception history of the subject intelligent agent i The input is fed into a multilayer perceptron network, and the local Q-value is calculated through the network's forward propagation. Then the agent starts from the local Q-value The action corresponding to the maximum Q value is selected from the corresponding action space to complete the entire decision-making process.

[0028] Preferably, the method further includes training all agents using a centralized training and distributed execution training paradigm.

[0029] As can be seen from the technical solutions provided by the embodiments of the present invention described above, the present invention significantly improves the task execution efficiency and success rate of multi-agent systems in dynamic and complex environments by modeling the active reasoning process of teammate agents and combining it with a dual screening mechanism to select modeling information, and by reasonably modeling teammate behavior and screening highly trustworthy collaborators.

[0030] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart of a multi-agent decision-making method based on active reasoning, provided as an embodiment of the present invention.

[0033] Figure 2 This is the active inference network described in the embodiments of the present invention.

[0034] Figure 3 This is the perspective conversion method described in the embodiments of the present invention.

[0035] Figure 4 This is the dual screening network described in the embodiments of the present invention. Detailed Implementation

[0036] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0037] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0038] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0039] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0040] This invention proposes a multi-agent decision-making method based on active reasoning. During the execution phase, this method utilizes only the agents' local observation information to perform a three-layer modeling of teammates (perception, belief, and action). First, a perception model is constructed to obtain the teammate's perception representation; second, a belief model is built to obtain the belief representation; and finally, an action model is constructed to obtain the action representation, thus effectively addressing the problem of missing teammate information. The method proposes a dual filtering mechanism: on the one hand, it selects high-confidence teammates based on the accuracy of perception representations to establish collaborative relationships; on the other hand, it introduces an attention mechanism to analyze the historical perception correlations between agents, achieving dynamic fusion of teammate belief representations and decision optimization.

[0041] The processing flow of a multi-agent decision-making method based on active reasoning proposed in this embodiment of the invention is as follows: Figure 1 As shown, the processing steps include the following:

[0042] Step S1: Construct a teammate-initiated reasoning network.

[0043] The teammate active reasoning network is the core component of the entire modeling method, responsible for modeling and predicting the behavioral logic of other agents. This network adopts a modular architecture design, containing three sub-networks: a perception modeling network, a belief modeling network, and an action prediction network, corresponding to the generation process of the teammate's perceptual representation, belief representation, and action representation, respectively.

[0044] In the perceptual modeling network, a coordinate transformation strategy is employed to construct a relative observation space. The position of each teammate is set as the origin, and the relative position information of other agents in this coordinate system is recalculated. The network processes the historical trajectory data of teammates through a gated recurrent unit (GRU) network, extracts temporal features, and fuses them with the current observation information to generate a perceptual representation containing spatiotemporal context.

[0045] The belief modeling network takes trajectory representations and teammate indices as inputs and generates a belief distribution through a learnable belief encoder. This design utilizes a Gaussian distribution to probabilistically model the intentions and goals of teammates, and obtains deterministic belief representations by sampling from the distribution through reparameterization techniques, effectively capturing the uncertainty of teammate agent behavior.

[0046] The action prediction network takes belief representation and historical perception information as input, integrates the two types of key information through feature concatenation operation, and uses a multi-layer perceptron (MLP) network to perform nonlinear transformation to generate the action distribution of teammates.

[0047] Step S2: The primary agent models the teammate's proactive reasoning process from three levels: perception, belief, and action. This yields the teammate's observational perception representation of the environment, its cognitive belief representation of the environment, and its predicted action for the next time step. These three aspects constitute a closed-loop cognitive process for the teammate, where the action representation can serve as posterior information feedback to adjust the earlier perception and belief representations. The modeling of the teammate's proactive reasoning process includes the following steps:

[0048] Step S21: Model the perception of teammates to obtain a perception representation. The method for modeling teammate perception employs a coordinate transformation strategy, setting the position of each teammate as the origin of the coordinate system, and recalculating the relative position information of other agents relative to that teammate based on this.

[0049] To illustrate this process more intuitively, let's take the StarCraft Multi-Agent Challenge (SMAC) as an example. Figure 2As shown, in a scenario involving two agents, the relative observation information of agent j can be obtained by transforming it from the perspective of agent i. Specifically, the relative coordinates of agent j are calculated. Regarding information filtering, only information within the overlapping field of view is retained. This information covers multiple key dimensions, including relative position, relative distance, health, shield value, and unit type, while any information outside the overlapping field of view is set to zero. To further refine the key information, only the observations of agent i are selected. Imaginary observation with teammate J The intersection of these elements is used to obtain an imagined perceptual portrait. Essentially, this process is a perspective shift operation that fuses and refines observational information from different agents, forming a more targeted information representation. This embodiment of the invention employs a Gated Recurrent Unit (GRU) network structure to extract the historical trajectory information of teammate j. After processing the historical trajectory information of teammate j through the GRU network, the trajectory representations of all teammates are further integrated.

[0050] Step S22: Model the teammate's beliefs to obtain a belief representation. For agent i, use its trajectory representation and the teammate's index id. -i To perform modeling, the trajectory representation of the main agent and the teammate index are fed into the belief encoder during the construction process to generate the belief distribution. in This represents the mean. The standard deviation is represented. The belief representation of teammates is obtained from the belief distribution using reparameters.

[0051] The primary agent uses its own historical trajectory information as a foundation, combined with the unique identifiers (ids) of teammate agents as indices, to obtain the belief distribution of each teammate agent at the current moment through a belief encoder. This belief distribution reflects the high-order probability estimate of the primary agent's intentions towards teammate agents. Further, deterministic belief representations are extracted from this distribution through sampling for subsequent action modeling.

[0052] Step S23: Model the teammate's actions to obtain an action representation. This embodiment of the invention concatenates the belief representation and the teammate's historical perception information to form a comprehensive set of input information. The belief representation includes key information such as the teammate's behavioral tendencies and internal beliefs, while the teammate's historical perception information records the teammate's observation data and state changes over a past period. By organically combining these two types of information, more comprehensive and richer input information can be provided to the action prediction network, thereby improving the accuracy and reliability of action prediction.

[0053] When modeling teammates' actions, the agent first fuses the teammate's perceptual representations obtained through perceptual modeling and the belief representations generated through belief modeling to form a comprehensive model of the teammate's state. Subsequently, a Multi-Layer Perceptron (MLP) network is used to model the actions of each teammate: the MLP takes the fused representation as input, passes it through fully connected layers and nonlinear transformations of activation functions, and finally outputs the probability distribution of the teammate's possible actions.

[0054] The agent takes the concatenated information as input and passes it to the action prediction network to obtain the imagined action distribution. This imagined action distribution is a probabilistic description of the actions a teammate might take. It reflects the likelihood of a teammate taking different actions under the current environment and historical information.

[0055] Step S3: Construct a dual-filtering network for modeling information.

[0056] Due to the inherent characteristics of local observability, errors are inevitably introduced during proactive reasoning. Indiscriminately using flawed teammate profiles will directly distort the agent's perception of the current scene, leading to non-cooperative behavior. This non-cooperative behavior not only reduces the overall efficiency of the system but may also disrupt the collaborative mechanisms in multi-agent systems, hindering the successful completion of tasks.

[0057] This invention presents a dual-filtering network for refined filtering and fusion of teammate modeling information. The network consists of two core modules: an accuracy filtering module and a relevance filtering module, which optimize the teammate modeling results from two dimensions: information quality and collaboration relevance, respectively.

[0058] In the accuracy screening module, the network uses a perceptual representation evaluation network to quantitatively assess teammates' perceptual representations. This evaluation network is based on a two-layer, multi-layer perceptron architecture and learns the mapping relationship from high-dimensional perceptual representations to scalar accuracy scores.

[0059] The relevance screening module is based on the attention mechanism and further integrates the highly reliable teammate belief representations after accuracy screening.

[0060] Step S4, Accuracy Filtering: The main agent filters the modeled teammate perception representations, filtering out unreliable information caused by observation errors, and selecting teammate indices with high accuracy as collaboration objects.

[0061] In this embodiment of the invention, the teammate's perceptual representation is generated through viewpoint transformation. However, due to the inherent characteristics of viewpoint transformation, the perceptual representation essentially only reflects a local viewpoint of the observation intersection, thus possessing incompleteness and potential bias. This incompleteness may negatively impact subsequent collaborative decision-making, therefore a robust evaluation method is needed to assess the accuracy of the perceptual representation. This embodiment of the invention proposes an accuracy scoring mechanism based on a mapping function, which learns a mapping function f:R h →R maps a teammate's perceptual representation to a scalar value, representing their accuracy score. At time step t, N×N perceptual representations are processed simultaneously to construct an evaluation matrix. Where N represents the number of agents. This represents the perception representation of agent i towards agent j. The accuracy score is specifically defined as:

[0062]

[0063] Where f(·) represents a two-layer multilayer perceptron network used to learn the mapping from perceptual representations to accuracy scores. ij The higher the value, the more accurate the perception representation of agent i is of agent j.

[0064] In this embodiment of the invention, the top_k indexes with the highest evaluation results are selected as the cooperation objects, and the belief representations of the teammate agents are further fused.

[0065] Step S5, Relevance Filtering: The agent fuses the belief representations of the selected collaborators. Building upon the accuracy filtering mechanism, to further improve collaboration efficiency from a decision-making perspective, a relevance filtering mechanism is also needed to select teammates more relevant to the current agent for collaboration. In multi-agent systems, collaboration often exhibits localized characteristics; therefore, not all teammates' information is equally important to the current decision.

[0066] The most intuitive approach to utilizing teammates' representations to aid decision-making is to combine their action representations to eliminate non-stationarity during execution. However, accurately modeling action representations is challenging due to limitations in local observability, especially when observation data is incomplete or contains errors. Therefore, this invention chooses to use belief representations instead of action representations. Belief representations, by capturing higher-level behavioral trends, can effectively mitigate the impact of single-step modeling errors, thus providing a more reliable basis for decision-making.

[0067] To achieve the fusion of relevance filtering and belief representation, this embodiment of the invention employs an attention mechanism. For agent i, its true perception history is... As a query, the historical perception representation of the selected k teammates is... As the key, the beliefs of these k teammates are represented. As a value, the relevance between each teammate's belief representation and the query of agent i is calculated through an attention mechanism, and the belief representations are weighted and fused, with an attention score α. ij The calculation formula is as follows:

[0068]

[0069] Among them, W Q and W K These are respectively applied to the query. s and keys The learnable weight matrix, d k This represents the dimension of the key vector. Attention score α i，j This represents the degree of correlation between agent i and teammate j. The higher the value, the more important teammate j is to agent i's current decision.

[0070] Step S6: The primary agent makes a decision based on the fusion representation of the teammate agents' beliefs and its own historical trajectory representation. The weighted fusion result based on the attention mechanism is represented as follows: in This is the fused belief feature vector, representing the weighted representation of agent i's belief in selected teammates. Subsequently, The real perceptual history of agent i They are collectively input into a multilayer perceptron network for calculating local Q-values.

[0071] Step S7: Train all agents using the Centralized Training with Decentralized Execution (CTDE) training paradigm. By modeling teammates with three layers of representations (perceptual representation, belief representation, and action representation) and employing a dual filtering mechanism (accuracy filtering and relevance filtering), this embodiment of the invention uses the CTDE framework based on centralized training and distributed execution to train the model.

[0072] In summary, this invention addresses the challenge of environmental non-stationarity in multi-agent reinforcement learning. By constructing a teammate active reasoning network and employing a dual-selection mechanism, it effectively enhances the system's ability to cope with complex dynamic environments. This method comprehensively models teammate behavior at three levels: perception, belief, and action. It selects highly reliable collaborators through accuracy screening and then fuses key belief representations through relevance screening, enabling the agent to more accurately predict teammate behavior patterns. This combination of dual-selection mechanism and active reasoning not only alleviates the environmental non-stationarity problem caused by partial observability and policy dynamics but also provides a reliable guarantee for efficient decision-making in complex collaborative tasks by multi-agent systems.

[0073] This invention innovatively combines a cognitive reasoning framework with the CTDE paradigm in its implementation. It models the psychological states (perceptions, beliefs, and actions) of teammates through an active reasoning network, fully leveraging the global information advantage of the CTDE framework during the intensive training phase. In the execution phase, a dual-screening mechanism selects highly reliable collaborators and integrates belief representations to achieve distributed decision-making. This method leverages the advantages of mental modeling in understanding teammates' intentions while inheriting the effectiveness of CTDE in mitigating non-stationarity, achieving a complementary advantage of both.

[0074] In collaborative decision-making scenarios involving drone swarms, the multi-agent decision-making method based on active reasoning proposed in this invention exhibits excellent adaptability. Each drone relies solely on its own onboard sensors to acquire local environmental information (such as the location of surrounding obstacles, the orientation and status of neighboring drones), achieving accurate cognition of teammates through three-layer modeling. In the perception modeling stage, a dynamic coordinate system transformation is employed: using the target teammate's drone position as a temporary origin, key parameters such as the three-dimensional coordinates, relative speed, and remaining battery power of other members within the swarm relative to this origin are recalculated, retaining only shared information within the coverage area of ​​the bidirectional sensors. The historical flight trajectories of the target teammate are fused using a bidirectional gated loop unit to generate a perception representation with spatiotemporal continuity. In the belief modeling stage, the main drone inputs its own navigation history data and the target teammate's aircraft ID into a belief encoder to generate a Gaussian distribution-based belief probability model, obtaining a deterministic belief representation through reparameterized sampling. In the action prediction stage, the aforementioned perception and belief representations are concatenated, and the probability distribution of the target teammate's future actions is output through a multilayer perceptron, forming a cognitive closed loop of "environmental observation → belief inference → behavior prediction".

[0075] To address the problem of perception error accumulation in complex airspace, a dual-screening mechanism plays a crucial optimization role. The accuracy screening module evaluates the credibility of all modeled teammates based on a two-layer perceptron: according to the accuracy score of the main UAV's perception representation of the target teammates, the UAV with the highest score is selected as the cooperation partner. The relevance screening module introduces an attention mechanism. The main UAV uses its own real perception sequence as the query vector, takes the perception representation of each cooperation partner as the key, and its belief representation as the value, dynamically assigns attention weights based on the airspace situation, and obtains an enhanced belief representation after weighted fusion. Finally, the main UAV inputs this fused belief feature along with its own historical trajectory into a Q-network to generate a real-time maneuvering strategy in a fully distributed architecture. This method significantly improves environmental adaptability in tasks such as UAV swarm adversarial and cooperative search. Through proactive reasoning, it accurately deconstructs the intentions of teammates, and the dual-screening mechanism effectively addresses perception noise in communication-denied environments, providing robust cooperation capabilities for distributed intelligent agent systems.

[0076] Furthermore, the technical solution of this invention possesses strong generalization ability and practical application value. Its modular design (perception modeling, belief modeling, action modeling) and universal screening mechanism (accuracy and relevance screening) enable it to adapt to intelligent agent systems of different sizes, dynamically changing environmental conditions, and diverse task requirements. Its design philosophy and implementation methods are also applicable to complex scenarios requiring multi-agent collaboration, such as military confrontation, autonomous driving, and robot cooperation. By rationally modeling teammate behavior and screening highly reliable collaborators, this invention significantly improves the task execution efficiency and success rate of multi-agent systems in dynamic and complex environments.

[0077] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0078] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0079] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0080] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multi-agent decision-making method based on active reasoning, characterized in that, include: A teammate active reasoning network is constructed. The teammate active reasoning network uses the agent to model the teammate active reasoning process from three levels: perception, belief and action. It obtains the teammate agent's observation and perception representation of the environment, the cognitive belief representation of the environment and the predicted action of the next time step. The main intelligent agent filters the perceptual representations of the modeled teammate intelligent agents, filtering out unreliable perceptual representations and their corresponding teammate intelligent agents, and retaining the teammate intelligent agents as the cooperation objects. The main intelligent agent fuses the belief representations of the selected collaborating objects and makes decisions based on the fusion result and its own historical trajectory representation.

2. The method according to claim 1, characterized in that, The aforementioned construction of a teammate proactive reasoning network utilizes the agent to model the teammate's proactive reasoning process from three levels: perception, belief, and action. This results in the teammate agent's observational perception representation of the environment, its cognitive belief representation of the environment, and its predicted action for the next time step, including: A teammate active reasoning network is constructed, comprising a perception modeling network, a belief modeling network, and an action prediction network. The perception modeling network uses a coordinate transformation strategy to construct a relative observation space, setting the position of each teammate agent as the origin, and recalculating the relative position information of other agents relative to that teammate agent. The belief modeling network takes trajectory representation and teammate index as input and generates a belief distribution through a learnable belief encoder. The action prediction network takes belief representation and historical perception information as input, integrates the two types of key information through feature concatenation, and uses a multilayer perceptron (MLP) network for nonlinear transformation to generate the action distribution of teammates. The agent modeled the teammate's active reasoning process from three levels: perception, belief, and action. This yielded the teammate's observational perception representation of the environment, its cognitive belief representation of the environment, and its predicted action for the next time step. The agent processed the teammate's historical trajectory data through a gated recurrent unit (GRU) network, extracted temporal features, and fused them with current observation information to generate a teammate's perception representation containing spatiotemporal context. The agent modeled each teammate's belief representation based on its own historical trajectory information and teammate ID index. Based on the modeling results of the teammate's perception and belief representations, the agent modeled the teammate's action at the current moment.

3. The method according to claim 2, characterized in that, The primary intelligent agent filters the perceptual representations of the modeled teammate intelligent agents, discarding unreliable perceptual representations and their corresponding teammate intelligent agents, and retaining the teammate intelligent agents as cooperation objects, including: A dual-filtering network is constructed, comprising an accuracy filtering module and a relevance filtering module. The accuracy filtering module is based on an accuracy scoring mechanism using a mapping function f:R. h →R maps the teammate agent's perceptual representation to a scalar value, which represents the teammate agent's accuracy score. At time step t, N×N perceptual representations are processed simultaneously to construct an evaluation matrix. Where N represents the number of agents. This represents the perception representation of agent i towards agent j. The accuracy score is specifically defined as: Where f(·) represents a two-layer multilayer perceptron network used to learn the mapping from perceptual representation to accuracy score, and c ij The higher the value, the more accurate the perception representation of agent i is to agent j. The top_k teammate agents with the highest accuracy scores are selected as the cooperation objects.

4. The method according to claim 3, characterized in that, The aforementioned intelligent agent fuses the belief representations of the selected collaborating objects, including: The correlation filtering module targets each teammate agent as a collaboration partner, and retrieves the actual perception history of teammate agent i. As a query, the historical perception representations of the selected k teammate agents are used. As the key, the beliefs of these k teammate agents are represented. As the value, the relevance between each teammate agent's belief representation and the query of teammate agent i is calculated through an attention mechanism, and the attention score α is used. ij The calculation formula is as follows: Among them, W Q and W K These are respectively applied to the query. s and keys The learnable weight matrix, d k This represents the dimension of the key vector. Attention score α i，j This indicates the degree of correlation between agent i and teammate j. The higher the value, the more important teammate j is to agent i's current decision. The belief representations of each collaborating object are weighted and fused. The weighted fusion result based on the attention mechanism is represented as follows: in Let represent the belief feature vector obtained by weighted fusion of the belief representations of the various teammate agents of the active agent i. This represents the belief of teammate agent j.

5. The method according to claim 4, characterized in that, The decision-making based on the fusion result and its own historical trajectory representation includes: The main intelligent agent i will The real perception history of the subject intelligent agent i The input is fed into a multilayer perceptron network, and the local Q-value is calculated through the network's forward propagation. Then the agent starts from the local Q-value The action corresponding to the maximum Q value is selected from the corresponding action space to complete the entire decision-making process.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes training all agents using a centralized training and distributed execution training paradigm.

Images