A method, device and equipment for transferable multi-agent reinforcement learning

Abstract

The present invention discloses a method, device and equipment for transferable multi-agent reinforcement learning, which includes: decomposing the common attributes of entities in a multi-agent environment to obtain a common entity class and corresponding entity attributes; defining and encoding the entity features based on the common entity class and the entity attributes to obtain an entity feature sequence; extracting and fusing the features of the entity feature sequence to obtain a fusion feature; decoupling the entity's actions based on the fusion feature to obtain entity actions including non-directional actions and directional actions; generalizing the fusion feature and the entity action in different hypothetical scenarios using a reinforcement learning algorithm to obtain a multi-agent model; and deploying the multi-agent model to the corresponding target hypothetical scenario to transfer knowledge. The learned knowledge can be transferred and applied more effectively, the dependence on a specific environment is reduced, and the generalization ability of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent reinforcement learning, and in particular to a transferable method, device and equipment for multi-agent reinforcement learning. Background Art

[0002] In the current application of multi-agent systems, reinforcement learning technology has become an important tool for solving complex decision-making problems. With the continuous expansion of application requirements, such as military confrontation games, machine cluster control, dynamic and complex environment management, and large-scale resource scheduling, the environment faced by multi-agent systems has become more complex and changeable. These scenarios require agents to quickly adapt to new environments and achieve efficient coordination and cooperation among multiple agents to ensure the optimization of the overall performance of the system. However, these application environments are usually highly dynamic, uncertain, and variable in scale, which brings severe challenges to the application of multi-agent reinforcement learning technology.

[0003] Traditional multi-agent reinforcement learning methods usually focus on fixed-size teams and relatively static environment designs, and their strategy learning relies on a large amount of interaction data and a long training cycle. This means that when faced with environmental changes or increases or decreases in the number of agents, multi-agent models often need to learn from scratch, lacking effective generalization capabilities and incremental training mechanisms, which severely limits the flexibility and efficiency of their practical applications. When the number of agents changes or the characteristics of the environment change significantly, existing multi-agent learning frameworks often have difficulty responding effectively.

[0004] First, due to the complex interactions between agents and the high nonlinearity of environmental feedback, it is difficult for the model to capture the key state characteristics and their interrelationships in different situations. The decision of each agent depends not only on its own state and strategy, but also on the behavior of other agents, resulting in complex dynamic behavior of the system as a whole. Existing learning methods are usually unable to accurately model these relationships when dealing with these complex interactions, resulting in insufficient effectiveness and adaptability of the strategy.

[0005] Secondly, the lack of an effective information integration mechanism makes it easy for agents to fall into the "curse of dimensionality" when dealing with high-dimensional state space, making it difficult to extract useful information that is conducive to decision-making. The high-dimensional state space brings huge computational challenges, making it difficult for agents to make high-quality decisions within limited time and computing resources. In this case, agents often find it difficult to effectively obtain and integrate information from the environment, reducing the quality and efficiency of decision-making. Therefore, it is of great significance to develop multi-agent reinforcement learning methods that can adapt to dynamic environments and changes in the number of agents. Summary of the invention

[0006] In view of this, the purpose of the present invention is to propose a transferable method, device and equipment for multi-agent reinforcement learning, aiming to solve the problems that the existing multi-agent reinforcement learning lacks effective generalization ability and incremental training mechanism, resulting in the need to relearn the agent model when the number of agents changes or the environment changes, affecting the flexibility and efficiency of practical applications.

[0007] To achieve the above object, the present invention provides a method for transferable multi-agent reinforcement learning, the method comprising:

[0008] Decompose the common attributes of entities in a multi-agent environment to obtain common entity classes and corresponding entity attributes;

[0009] Based on the general entity class and the entity attributes, the entity is characterized and encoded to obtain an entity feature sequence;

[0010] Extracting and fusing features of the entity feature sequence to obtain fused features;

[0011] Decoupling the entity's actions based on the fusion features to obtain entity actions including non-directional actions and directional actions;

[0012] Using a reinforcement learning algorithm to generalize the fusion features and the entity actions in different scenarios to obtain a multi-agent model;

[0013] The multi-agent model is deployed to the corresponding target scenario to transfer knowledge.

[0014] Preferably, the general attributes of entities in the multi-agent environment are decomposed to obtain general entity classes and corresponding entity attributes, including:

[0015] Extract the common attributes and behaviors of all entities from the multi-agent environment, create the general entity class according to the extracted attributes and behaviors, and initialize the corresponding entity attributes; the entity attributes include state attributes, action capabilities, environmental interactions, dynamic attributes, and identity identification; the state attributes include the current state information of the entity; the action capabilities include the entity's mobility, attack methods, and resource collection capabilities; the environmental interactions include physical collision rules and object manipulation capabilities; the dynamic attributes include the entity's experience accumulation, skill upgrades, and emotional state changes; the identity identification is a unique identity representation when calculating interactions between entities.

[0016] Preferably, the entity feature definition and encoding based on the general entity class and the entity attribute to obtain the entity feature sequence includes:

[0017] Creating a simulation entity that inherits from the general entity class and determining scenario details;

[0018] Based on the scenario details, the simulation entities are divided into different categories to obtain a plurality of simulation entity classes;

[0019] Determine the basic attributes corresponding to the simulation entity class, and update the entity attributes of each simulation entity according to the basic attributes;

[0020] Quantitatively and qualitatively processing the entity attributes to obtain entity features;

[0021] The entity features are encoded by a preset encoding method to obtain a plurality of feature vectors, and the plurality of feature vectors are spliced ​​by a sequence splicing method to obtain the entity feature sequence.

[0022] Preferably, the preset encoding method includes normalization and One-hot; encoding the entity feature by the preset encoding method includes:

[0023] The normalization is used to convert the continuous features in the entity features into a preset numerical range, and the One-hot is used to convert the discrete features in the entity features into a binary column, wherein each column represents a discrete value; the continuous features include health and cost; the discrete features include weapon category and color.

[0024] Preferably, before extracting and fusing the entity feature sequence to obtain the fused feature, the method further includes:

[0025] The derived features of each entity in a dynamic environment are calculated according to the entity feature sequence; the derived features and the own entity features, friendly entity features, and enemy entity features in the entity feature sequence are combined, and after the combination, the feature vectors of each entity are vertically spliced ​​to obtain a feature matrix.

[0026] Preferably, extracting and fusing features of the entity feature sequence to obtain fused features includes:

[0027] Performing a linear transformation on the feature matrix through a self-attention mechanism to obtain a query vector, a key vector, and a value vector;

[0028] Calculating the similarity between the query vector and all the key vectors to obtain a series of original weights, and normalizing the original weights using a softmax function to obtain weighting coefficients;

[0029] The value vector is weighted and summed according to the weight coefficient to generate the fusion feature.

[0030] Preferably, the decoupling of the entity's actions based on the fusion features to obtain entity actions including non-directional actions and directional actions includes:

[0031] Mapping the fused features to the corresponding entity, and inputting the first dimension feature in the fused features into the strategy network to calculate the probability or q value of the action, to obtain the non-directional action of the entity;

[0032] The entity features corresponding to the other dimensional features in the fusion features are input into the strategy network to calculate the probability or q value of the action, so as to obtain the directional action of the entity.

[0033] Preferably, the reinforcement learning algorithm includes a learner module and an actor module; the reinforcement learning algorithm is used to generalize the fusion features and the entity actions in different scenarios to obtain a multi-agent model, including:

[0034] During the training process, the fused features are input into the learner module for synchronous training to update its own parameters, the fused features and the entity actions are input into the actor module for action decoupling, and a hypothetical scenario is randomly selected for training to update the model parameters. When the reward value or the winning rate reaches a preset value, the next hypothetical scenario is switched to continue training to obtain the multi-agent model.

[0035] To achieve the above object, the present invention further provides a multi-agent reinforcement learning transferable device, the device comprising:

[0036] The entity layer building module is used to disassemble the common attributes of entities in the multi-agent environment to obtain common entity classes and corresponding entity attributes;

[0037] A simulation layer construction module, used for defining and encoding entity features based on the general entity class and the entity attributes to obtain an entity feature sequence;

[0038] A self-attention feature module is used to extract and fuse features of the entity feature sequence to obtain fused features;

[0039] A self-attention action module, used for decoupling the action of the entity based on the fusion features to obtain the entity action including non-directional action and directional action;

[0040] A reinforcement learning operator module, used for generalizing the fusion features and the entity actions in different scenarios using a reinforcement learning algorithm to obtain a multi-agent model;

[0041] The migration module is used to deploy the multi-agent model to the corresponding target scenario to transfer knowledge.

[0042] In order to achieve the above objectives, the present invention also proposes a multi-agent reinforcement learning transferable device, including a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a multi-agent reinforcement learning transferable method as described in the above embodiment.

[0043] In order to achieve the above objectives, the present invention also proposes a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the steps of a transferable method for multi-agent reinforcement learning as described in the above embodiment.

[0044] In order to achieve the above objectives, the present invention also proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of a transferable method for multi-agent reinforcement learning as described in the above embodiment.

[0045] Beneficial effects:

[0046] The above scheme, by disassembling the common attributes of entities in the multi-agent environment and defining and encoding the features of the entities based on these attributes, can build a more general and flexible multi-agent model, so that the model can more effectively transfer and apply the learned knowledge when facing different hypothetical scenarios, reduce the dependence on specific environments, and improve the generalization ability of the model; by using reinforcement learning algorithms to perform generalization training in different hypothetical scenarios, the model can adapt to various environmental changes and the increase or decrease in the number of agents, dynamically adjust the dimensions of agent actions, flexibly increase or decrease the focus on certain specific actions, optimize collaborative operations and confrontation strategies, and enhance the flexibility and adaptability of the model in practical applications.

[0047] The above scheme extracts and fuses features from entity feature sequences, and introduces a self-attention mechanism that allows the agent to dynamically focus on the most critical information, thereby achieving effective screening and fusion of environmental features and peer status. It can more accurately capture key state features and their interrelationships in different situations, thereby improving the effectiveness and accuracy of decision-making, and enhancing the ability of the agent model to quickly adapt to external changes.

[0048] The above scheme decouples the actions between agents based on the self-attention mechanism, obtains non-directional actions and directional actions, dynamically adjusts the dimensions of agent actions, flexibly enhances or weakens the attention to certain specific actions, optimizes collaborative operations and confrontation strategies, and makes the selection of actions more refined and flexible, which helps agents make more appropriate decisions in complex environments.

[0049] The above scheme, through normalization and One-hot encoding, as well as effective processing of high-dimensional state space (such as the generation of feature matrices and the application of self-attention mechanisms), can reduce computational complexity and improve training efficiency while ensuring decision quality. By utilizing the collaboration of the learner module and the actor module, and switching training strategies in different scenarios, the model can achieve better training results within limited time and computing resources; making subsequent maintenance and expansion more convenient and efficient, and can easily add new entity types or adjust model parameters according to specific application requirements to meet different application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flowchart of a transferable method for multi-agent reinforcement learning provided by one embodiment of the present invention.

[0052] Figure 2 A schematic diagram of the structure of a multi-agent system provided in one embodiment of the present invention.

[0053] Figure 3 A schematic diagram of the structure of a transferable device for multi-agent reinforcement learning provided in one embodiment of the present invention.

[0054] The realization of the purpose, functional features and advantages of the invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0056] The content of the present invention is described in detail below with reference to embodiments.

[0057] Reference Figure 1 The figure is a flow chart of a transferable method of multi-agent reinforcement learning provided by one embodiment of the present invention.

[0058] In this embodiment, the method includes:

[0059] S11, decompose the common attributes of entities in the multi-agent environment to obtain common entity classes and corresponding entity attributes.

[0060] Furthermore, in step S11, the common attributes of the entities in the multi-agent environment are decomposed to obtain common entity classes and corresponding entity attributes, including:

[0061] Extract the common attributes and behaviors of all entities from a multi-agent environment (an environment in which multiple agents (i.e., entities that can make decisions autonomously) interact with each other, which can be static or dynamic), create the general entity class based on the extracted attributes and behaviors, and initialize the corresponding entity attributes; the entity attributes include state attributes, action capabilities, environmental interactions, dynamic attributes, and identity identifiers; the state attributes include the current state information of the entity; the action capabilities include the entity's mobility, attack methods, and resource collection capabilities; the environmental interactions include physical collision rules and object manipulation capabilities; the dynamic attributes include the entity's experience accumulation, skill upgrades, and emotional state changes; the identity identifier is a unique identity representation when computing interactions between entities.

[0062] Reference Figure 2As shown, in this embodiment, the method is implemented based on a multi-agent system, which includes a physical layer, a simulation layer, a self-attention feature module, a self-attention action module, and a reinforcement learning operator module. By constructing the physical layer and the simulation layer, different types of entities and their feature representations can be flexibly defined and configured, providing a unified framework for the multi-agent environment and simplifying the process of building a multi-agent system. Among them, the physical layer is responsible for disassembling the common attributes of each entity in the multi-agent environment (such as Figure 2 The multi-agent military exercise simulation system shown in the figure decomposes the common attributes of entities such as infantry, tanks, fighters, drones, ships and infrastructure to obtain dynamic attributes such as position, speed, and ammunition, action capabilities such as movement, attack, and reconnaissance, environmental interactions such as terrain adaptability and concealment, and identity identification such as troop numbers and task roles. ), capturing the uniqueness of entities and standardizing the representation of different types of entity features, which not only improves the expression ability of agent behavior strategies, but also promotes the refined management and coordination of agent behaviors, ensuring the effectiveness and adaptability of strategies in complex scenarios; the simulation layer is responsible for the management of entities, including entity feature definition, feature encoding, action mapping, scenario setting (setting specific tasks and environments, such as defense, attack, reconnaissance, etc.) and reward design (designing reward mechanisms based on task completion and combat effectiveness, such as destroying enemy entities, successful reconnaissance, etc.); self-attention feature module (with Figure 2 The self-attention action module (corresponding to the feature module in Figure 2 The reinforcement learning operator module accesses the features processed by the feature module and the actions processed by the action module, uses the reward value returned by the simulation layer for training, and returns the actions to the simulation layer for action mapping.

[0063] Specifically, the process of disassembling common attributes is mainly to define entities, disassemble and extract the common attributes and behaviors of all entities from the multi-agent environment, create a common entity class and initialize entity attributes. Entity attributes include state attributes, action capabilities, environmental interactions, dynamic attributes, and identity identification. Among them, state attributes include the current state information of the entity, such as position, speed, health value, endurance, etc., which are the basis for determining entity behavior and interaction; action capabilities describe the range and limitations of actions that an entity can perform, such as mobility, attack methods, and the ability to collect resources; environmental interactions reflect the way entities interact with other parts of the environment (including static objects and other entities), such as physical collision rules and object manipulation capabilities; dynamic attributes refer to attributes that change over time, such as experience accumulation, skill upgrades, emotional state changes, etc. These attributes reflect the evolution of entities; identity identification is the only representation when calculating interactions between entities. The above representation of entity attributes can not only capture the uniqueness of each entity, but also realize the transfer of knowledge between different tasks or scenarios.

[0064] In military exercises, the multi-agent military exercise simulation system can flexibly configure and manage various combat entities. For example:

[0065] Entities of our forces:

[0066] Infantry: responsible for close combat and defending positions; tanks: provide powerful fire support and cooperate with infantry; fighters: perform air support missions and attack ground and air targets; drones: perform reconnaissance and surveillance missions and provide real-time intelligence; ships: perform maritime combat missions and protect maritime transport lines.

[0067] Enemy Force Entities:

[0068] Infantry: carry out raids and occupation missions; tanks: carry out armored assaults and destroy our positions; fighter jets: carry out air superiority and bombing missions; drones: carry out reconnaissance and electronic jamming missions.

[0069] At the beginning of the exercise, our side and the enemy side each deployed 50 infantry, 20 tanks, 10 fighter jets, 15 drones, and 5 ships; as the exercise progressed, the number of enemy infantry increased to 100, while the number of our drones decreased to 10. The self-attention feature module, self-attention action module, and reinforcement learning operator module are processed as follows:

[0070] Self-attention feature module: processes entity feature sequences of various friendly and enemy entities, extracts dynamic attributes and action capability features of infantry, tanks, fighters, drones and ships, etc.;

[0071] When the number of enemy infantry increases to 100, only the length of the entity feature sequence increases. The self-attention feature module does not need to be adjusted or retrained, and can automatically process the features of the newly added infantry and merge them with the features of other entities.

[0072] When processing the features of our drones, the number of our drones is reduced from 15 to 10, and the self-attention feature module can still efficiently extract and fuse the drone features.

[0073] Self-Attention Action Module: As the number of enemy infantry increases and the number of our drones decreases, the self-attention action module can flexibly decouple actions, increase strike actions and reduce drone entity actions, ensuring that each entity executes the strategy efficiently.

[0074] Reinforcement learning operator module: There is no need to relearn the entire model. Instead, based on the features and actions of the self-attention feature module and the self-attention action module, the strategy is adjusted in real time to handle the dynamic changes brought about by the changes in the number of intelligent agents. Under the reward mechanism, by continuously optimizing the intelligent agent behavior strategy, it ensures that each combat entity can still perform tasks efficiently in a dynamic environment.

[0075] In a simulated military exercise, the system needs to conduct comprehensive simulations of different types of combat entities. When the environment changes, for example, the enemy forces suddenly increase the number of drones, the system can make effective adjustments and responses without relearning. This significantly improves the flexibility and efficiency of the system and ensures the accuracy and reliability of the simulation results.

[0076] S12, defining and encoding entity features based on the general entity class and the entity attributes to obtain an entity feature sequence.

[0077] Furthermore, in step S12, the entity is characterized and encoded based on the general entity class and the entity attribute to obtain an entity feature sequence, including:

[0078] S12-1, creating a simulation entity inherited from the general entity class and determining the scenario details;

[0079] S12-2, based on the assumption details, dividing the simulation entities into different categories to obtain a plurality of simulation entity classes;

[0080] S12-3, determining basic attributes corresponding to the simulation entity class, and updating the entity attributes of each simulation entity according to the basic attributes;

[0081] S12-4, performing quantitative and qualitative processing on the entity attributes to obtain entity features;

[0082] S12-5, encoding the entity features by a preset encoding method to obtain a plurality of feature vectors, and concatenating the plurality of feature vectors by a sequence concatenation method to obtain the entity feature sequence.

[0083] Further, in step S12-5, the preset encoding method includes normalization and One-hot; encoding the entity feature by the preset encoding method includes:

[0084] The normalization is used to convert the continuous features in the entity features into a preset numerical range, and the One-hot is used to convert the discrete features in the entity features into a binary column, wherein each column represents a discrete value; the continuous features include health and cost; the discrete features include weapon category and color.

[0085] In this embodiment, the simulation layer is constructed to manage entities, including entity feature definition and feature coding, entity action mapping, and entity reward design; it includes simulation entities and scenario details. The simulation entity inherits from the general entity class and is divided according to categories. The scenario details are the application scenarios of the intelligent agent. Specifically, it includes: determining the scenario details, and splitting the entity details from the scenario details (the entity details are the functions, behavior patterns, and interaction methods of the simulation entity). Based on the entity details, the simulation entity is divided into different categories according to the functions, behavior patterns, and interaction methods of the simulation entity, and inherits the general entity class to obtain multiple simulation entity classes.

[0086] The process of defining the characteristics of an entity includes: determining basic attributes from the simulation entity, updating the entity attributes of each simulation entity, and defining key attributes for different types of simulation entities; designing a feature set (the feature set can be understood as a set of variables used to describe the multi-dimensional characteristics of an entity) to complete the abstract representation of the entity attributes, which can be quantitative (numerical, such as health, cost) or qualitative (category, such as weapon category, color), or composite (such as geographic location coordinates), covering all aspects from basic attributes (usually referring to the inherent and relatively static characteristics of an entity, such as a product's brand, model, production date, etc., which can be directly obtained from the data set without complex calculations or inferences) to advanced states (involving deeper levels, dynamic changes, or information that needs to be inferred, such as a product's loss status, occupancy rate, etc.), and obtaining entity characteristics through quantitative and qualitative processing of entity attributes.

[0087] The process of encoding the entity features includes: effectively encoding the defined entity features through normalization and One-Hot encoding methods to obtain multiple feature vectors, and using sequence splicing to treat the encoded feature vectors as elements in a sequence to ensure the universality and dynamics of the entity feature sequence. Specifically, through normalization, continuous features (such as health and cost) are converted into a unified numerical range to eliminate the impact of the dimensions and numerical ranges between different features; through One-hot, discrete features (such as weapon category and color) are converted into a binary column, each column represents a discrete value, thereby avoiding the qualitative numerical relationship from being mistakenly interpreted as a size relationship; multiple feature vectors obtained through encoding are spliced ​​to form the final entity feature sequence, ensuring that the data input to the model is in a consistent format and suitable for algorithm processing.

[0088] The entity action mapping process includes: in the simulation layer, abstract action concepts are mapped into specific operation instructions to ensure that the intelligent agent can perform a series of complex behaviors. By defining all possible actions that different simulation entities can perform, considering the effectiveness and feasibility of the actions according to the simulation entity class, and mapping the action codes output by the reinforcement learning algorithm to the simulation action instructions, it incorporates the response strategy to the dynamic changes of the environment and enhances the adaptability of the intelligent agent in changing scenarios.

[0089] The process of entity reward design includes: the simulation layer adopts a fine-tuning mechanism, which aims to guide the agent to learn the optimal strategy through positive incentives and negative feedback. The reward function is configurable and allows adjustment according to specific learning goals or environmental challenges, thereby maintaining learning efficiency while promoting the generalization and migration of strategies. In addition, by designing a multi-level and multi-dimensional reward function, the agent is encouraged to learn more sophisticated and coordinated behavior strategies in complex tasks; the multi-level and multi-dimensional reward function includes a primary reward layer, an intermediate strategy layer, and a high-level goal layer. Among them, the primary reward layer rewards the basic action feedback corresponding to the agent to guide the agent to complete the basic action, that is, to give immediate feedback for the basic action or simple task completed by the agent to enhance its direct response to the environment; the intermediate strategy layer rewards effective strategies and guides the agent to complete short-term tasks, that is, to set rewards for the effective strategies and collaborative behaviors shown by the agent in the medium and short-term goals, and promote the coordination and optimization of strategies between agents; the high-level goal layer rewards the long-term rewards of task planning and guides the agent to complete the assumed task, that is, to set long-term rewards closely related to the final task success to ensure that the agent's behavior points to the optimization of the overall task. Through this hierarchical reward mechanism, not only are agents encouraged to maximize their individual capabilities, but it also promotes the emergence of efficient cooperation strategies in multi-agent systems.

[0090] S13, extracting and fusing features of the entity feature sequence to obtain fused features.

[0091] Furthermore, before extracting and fusing features of the entity feature sequence to obtain fused features, the method further includes:

[0092] The derived features of each entity in a dynamic environment are calculated according to the entity feature sequence; the derived features and the own entity features, friendly entity features, and enemy entity features in the entity feature sequence are combined, and after the combination, the feature vectors of each entity are vertically spliced ​​to obtain a feature matrix.

[0093] Furthermore, in step S13, extracting and fusing features of the entity feature sequence to obtain fused features includes:

[0094] S13-1, linearly transform the feature matrix through a self-attention mechanism to obtain a query vector, a key vector, and a value vector;

[0095] S13-2, calculating the similarity between the query vector and all the key vectors to obtain a series of original weights, and normalizing the original weights using a softmax function to obtain weighting coefficients;

[0096] S13-3, performing weighted summation on the value vector according to the weighting coefficient to generate the fusion feature.

[0097] In this embodiment, by introducing a self-attention mechanism to process entity feature sequences, each agent can dynamically focus on the most critical environmental features or other entity states for its current decision, and realize the fusion of feature vectors, integrate the global context information shared by the agents, and improve decision-making efficiency and adaptability. Specifically, the derived features of each entity in a dynamic environment (referring to a scenario where the state of the environment changes over time, and this change may be caused by external factors or by the actions of the agent itself. In a dynamic environment, the agent needs to be able to adapt to these changes and make corresponding decisions) are calculated according to the entity feature sequence, such as distance, relative speed, and collaborative indicators; the features of the entity itself, the friendly entity features, the enemy entity features, and the derived features between entities are combined, wherein the features of the entity itself, the friendly entity features, and the enemy entity features are all from the entity feature sequence; after the combination, the feature vectors of each entity are vertically spliced ​​into a feature matrix as the input of the self-attention feature module, and the features are extracted and fused using a multi-layer self-attention mechanism. Expressed in mathematical form, assuming there are n entities, the entity features have m dimensions after encoding, and the derived features derived from the entities have k dimensions. Then the input entity feature sequence is a feature vector with a dimension of n*m, and the combined feature is a feature matrix with a dimension of (n, n*(m+k)), that is, the rows represent different entities, and the columns represent the combined features of the entity.

[0098] After the feature combination is completed, the multi-layer self-attention mechanism is used to extract and fuse the features. The core of the self-attention mechanism is to calculate the attention weight. The feature vector of each entity can interact with the feature vectors of other entities to capture global dependencies. Through the self-attention mechanism, attention weights are automatically assigned to the feature vectors observed by each entity in a dynamic environment. This mechanism ensures that the intelligent agent can focus on analyzing the most critical information according to the current decision-making needs. The feature matrix is ​​linearly transformed, that is, at each time step, each element (feature vector) in the entity feature sequence is converted into a high-dimensional vector representation of query, key, and value. This step is completed by a deep neural network, which enhances the model's ability to understand the interaction between complex states. The dot product attention mechanism is used to calculate the similarity between the query vector and all key vectors to obtain a series of original weights; these original weights are normalized using the softmax function to obtain the weighting coefficient A, ensuring that the sum of all weights is 1, and that features with high importance obtain higher weight values, thereby effectively highlighting the information that contributes most to the decision. Based on the weighted coefficients calculated above, the value vector is weighted and summed to generate a fused feature (a comprehensive context vector), represented by Z=AV. This context vector summarizes the key environmental features and entity state information after being filtered by the self-attention mechanism, providing the agent with a highly concentrated and decision-related global perspective. Through the multi-layer self-attention mechanism, each layer repeats the above process, extracts and fuses features layer by layer, and after the last layer of self-attention mechanism, the feature matrix Z obtained is the fused feature. Assuming that the encoding dimension of the attention mechanism is h, the dimension of the fused feature Z is (n, h). The feature dimension of the entity is encoded from the original n*(m+k) to h by the attention mechanism, effectively fusing the features between different entities. At this time, the features of the first dimension (0, h) are taken as the feature input to the reinforcement learning operator module.

[0099] S14, decoupling the entity's actions based on the fusion features to obtain entity actions including non-directional actions and directional actions.

[0100] Furthermore, in step S14, the entity action is decoupled based on the fusion feature to obtain entity actions including non-directional actions and directional actions, including:

[0101] S14-1, mapping the fused features to the corresponding entity, and inputting the first dimension feature in the fused features into the strategy network to calculate the probability or q value of the action, to obtain the non-directional action of the entity;

[0102] S14-2, input the entity features corresponding to the other dimensional features in the fusion features into the strategy network to calculate the probability or q value of the action, and obtain the directional action of the entity.

[0103] In this embodiment, the fusion feature is mapped to the corresponding entity, that is, the features of each dimension are mapped to the entity, and the first dimension feature (0, h) is used as input by connecting to the existing reinforcement learning operator strategy network. The strategy network calculates the probability or q value of the action (q value or value, which is an indicator used to measure the action) to obtain the non-directional action of the entity; the entity features corresponding to the remaining other dimensional features (1:n, h) are input into the strategy network to calculate the probability or q value of the action, and the directional action of the entity is obtained, that is, the action of linking with the friendly entity or confronting the enemy entity. Among them, the directional action refers to the basic action of the entity, and the non-directional action refers to the action derived from the entity and other entities.

[0104] S15, using a reinforcement learning algorithm to perform generalization training on the fusion features and the entity actions in different hypothetical scenarios to obtain a multi-agent model.

[0105] Further, in step S15, the reinforcement learning algorithm includes a learner module and an actor module; the reinforcement learning algorithm is used to generalize the fusion features and the entity actions in different scenarios to obtain a multi-agent model, including:

[0106] During the training process, the fused features are input into the learner module for synchronous training to update its own parameters, the fused features and the entity actions are input into the actor module for action decoupling, and a hypothetical scenario is randomly selected for training to update the model parameters. When the reward value or the winning rate reaches a preset value, the next hypothetical scenario is switched to continue training to obtain the multi-agent model.

[0107] In this embodiment, by combining with advanced reinforcement learning algorithms, the operator learner module and actor module are mainly connected to adjust features and actions. Among them, the learner module is connected to the self-attention feature module to receive the input of fused features and synchronously train and update its own parameters (referring to updating the parameters of the self-attention feature module); the actor module is connected to the self-attention feature module and the self-attention action module to receive the fused features and entity action inputs to achieve action decoupling. The reinforcement learning algorithm is used to integrate the observation values ​​and decouple the actions of the intelligent agent. On the one hand, it highlights the high-value information that is particularly critical to strategy learning. On the other hand, it ensures that the strategy can quickly respond to local environmental changes when updating, and maintain the consistency and coherence of the overall strategic direction. It improves the learning efficiency and strategy performance of the multi-agent system in the face of dynamic and uncertain environments, realizes the effective learning and iterative update of strategies, and provides a strong framework support for solving complex collaborative tasks.

[0108] After the reinforcement learning algorithm is connected to the self-attention feature module and the self-attention action module, the changes in the features and action dimensions caused by the changes in the number of scenes and entities will not affect the calculation of the algorithm. Therefore, generalization training is performed by switching the assumption scenarios during the training process to enhance the perception and understanding ability of the feature module. First, it is necessary to collect and annotate a variety of different assumption scenarios as training scenarios, including different environmental settings, task objectives and constraints, and standardize the features in each scenario. During the training process, a scene is randomly selected for training, and the model parameters (including the parameters of the self-attention feature module and the parameters of the reinforcement learning algorithm) are updated until the reward value or the winning rate reaches a certain value, and then the next assumption scenario is switched to repeat this process, and finally a multi-agent model is obtained. In addition, cross-validation is used to evaluate the performance of the model in different assumption scenarios, and hyperparameters are adjusted and regularization techniques are introduced to prevent overfitting. Through this process, a model with strong generalization ability can be constructed, which can stably and accurately perform feature perception and action decision-making in a variety of different assumption scenarios.

[0109] S16, deploying the multi-agent model to the corresponding target scenario to transfer knowledge.

[0110] In this embodiment, the multi-agent model obtained through generalization training can be quickly deployed to any similar scenario to achieve migration application; for situations where there are major changes in strategy, the self-attention feature module can be migrated, and the self-attention feature module can share the representation space between different tasks to promote cross-domain migration of knowledge. By incorporating a large amount of multi-domain data in the previous training stage and building a feature representation library rich in generalization capabilities, the intelligent agent can quickly identify and utilize useful information in the new environment under this rich knowledge framework, and even when facing unseen tasks or environmental challenges, it can significantly shorten the exploration period and improve learning efficiency.

[0111] Reference Figure 3 Shown is a schematic diagram of the structure of a multi-agent reinforcement learning transferable device provided by an embodiment of the present invention.

[0112] In this embodiment, the device 20 includes:

[0113] The entity layer construction module 21 is used to disassemble the common attributes of entities in the multi-agent environment to obtain common entity classes and corresponding entity attributes;

[0114] A simulation layer construction module 22, used to define and encode the entity features based on the general entity class and the entity attributes to obtain an entity feature sequence;

[0115] A self-attention feature module 23 is used to extract and fuse features of the entity feature sequence to obtain fused features;

[0116] A self-attention action module 24, configured to decouple the action of the entity based on the fusion features to obtain entity actions including non-directional actions and directional actions;

[0117] A reinforcement learning operator module 25, for generalizing the fusion features and the entity actions in different scenarios using a reinforcement learning algorithm to obtain a multi-agent model;

[0118] The migration module 26 is used to deploy the multi-agent model to the corresponding target scenario to transfer knowledge.

[0119] Each unit module of the device 20 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the above corresponding steps for details.

[0120] The embodiment of the present invention also provides a multi-agent reinforcement learning transferable device, the device comprising the multi-agent reinforcement learning transferable device as described above, wherein the multi-agent reinforcement learning transferable device can be used Figure 3 The structure of the embodiment can be executed accordingly. Figure 1The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0121] The device includes: a mobile phone, a digital camera, a tablet computer, or other devices with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0122] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0123] The input unit can be used to receive input digital or character or image information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment includes not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0124] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel, and optionally, the display panel may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface may cover the display panel, and when the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event, and then the processor provides a corresponding visual output on the display panel according to the type of touch event.

[0125] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not installed in a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The multi-agent reinforcement learning transferable method shown. The computer-readable storage medium can be a read-only memory, a disk or an optical disk, etc.

[0126] The embodiment of the present invention also provides a computer program product, including a computer program / instruction, which is loaded and executed by a processor to implement Figure 1 A transferable method for multi-agent reinforcement learning is shown.

[0127] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, equipment embodiment and storage medium embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0128] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0129] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A transferable method for multi-agent reinforcement learning, It is characterized in that The method is applied to a multi-agent military exercise simulation system; the method comprises: The common attributes of entities in a multi-agent environment are disassembled to obtain common entity classes and corresponding entity attributes; further comprising: disassembling common attributes of entities including infantry, tanks, fighters, drones, ships and infrastructure to obtain: dynamic attributes including position, speed and ammunition, action capabilities including movement, attack and reconnaissance, environmental interactions including terrain adaptability and concealment, and identification of troop numbers and mission roles; Based on the general entity class and the entity attributes, the entity is characterized and encoded to obtain an entity feature sequence; Extracting and fusing the entity feature sequence to obtain fused features; further comprising: extracting dynamic attributes and action capability features of infantry, tanks, fighters, drones and ships; Decoupling the entity's actions based on the fusion features to obtain entity actions including non-directional actions and directional actions; Using a reinforcement learning algorithm to generalize the fusion features and the entity actions in different scenarios to obtain a multi-agent model; The multi-agent model is deployed to the corresponding target scenario to transfer knowledge.

2. A transferable method for multi-agent reinforcement learning according to claim 1, It is characterized in that The general attributes of entities in the multi-agent environment are decomposed to obtain general entity classes and corresponding entity attributes, including: Extract the common attributes and behaviors of all entities from the multi-agent environment, create the general entity class according to the extracted attributes and behaviors, and initialize the corresponding entity attributes; the entity attributes include state attributes, action capabilities, environmental interactions, dynamic attributes, and identity identification; the state attributes include the current state information of the entity; the action capabilities include the entity's mobility, attack methods, and resource collection capabilities; the environmental interactions include physical collision rules and object manipulation capabilities; the dynamic attributes include the entity's experience accumulation, skill upgrades, and emotional state changes; the identity identification is a unique identity representation when calculating interactions between entities.

3. A transferable method for multi-agent reinforcement learning according to claim 1, It is characterized in that The step of defining and encoding the entity features based on the general entity class and the entity attributes to obtain an entity feature sequence includes: Creating a simulation entity that inherits from the general entity class and determining scenario details; Based on the scenario details, the simulation entities are divided into different categories to obtain a plurality of simulation entity classes; Determine the basic attributes corresponding to the simulation entity class, and update the entity attributes of each simulation entity according to the basic attributes; Quantitatively and qualitatively processing the entity attributes to obtain entity features; The entity features are encoded by a preset encoding method to obtain a plurality of feature vectors, and the plurality of feature vectors are spliced ​​by a sequence splicing method to obtain the entity feature sequence.

4. A transferable method for multi-agent reinforcement learning according to claim 3, It is characterized in that The preset encoding method includes normalization and One-hot; encoding the entity feature by the preset encoding method includes: The normalization is used to convert the continuous features in the entity features into a preset numerical range, and the One-hot is used to convert the discrete features in the entity features into a binary column, wherein each column represents a discrete value; the continuous features include health and cost; the discrete features include weapon category and color.

5. A transferable method for multi-agent reinforcement learning according to claim 1, It is characterized in that Before extracting and fusing features of the entity feature sequence to obtain fused features, the method further includes: The derived features of each entity in a dynamic environment are calculated according to the entity feature sequence; the derived features and the own entity features, friendly entity features, and enemy entity features in the entity feature sequence are combined, and after the combination, the feature vectors of each entity are vertically spliced ​​to obtain a feature matrix.

6. A transferable method for multi-agent reinforcement learning according to claim 5, It is characterized in that The extracting and fusing features of the entity feature sequence to obtain fused features includes: Performing a linear transformation on the feature matrix through a self-attention mechanism to obtain a query vector, a key vector, and a value vector; Calculating the similarity between the query vector and all the key vectors to obtain a series of original weights, and normalizing the original weights using a softmax function to obtain weighting coefficients; The value vector is weighted and summed according to the weight coefficient to generate the fusion feature.

7. A transferable method for multi-agent reinforcement learning according to claim 1, It is characterized in that The decoupling of the entity's actions based on the fusion features to obtain entity actions including non-directional actions and directional actions includes: Mapping the fused features to the corresponding entity, and inputting the first dimension feature in the fused features into the strategy network to calculate the probability or q value of the action, to obtain the non-directional action of the entity; The entity features corresponding to the other dimensional features in the fusion features are input into the strategy network to calculate the probability or q value of the action, so as to obtain the directional action of the entity.

8. A transferable method for multi-agent reinforcement learning according to claim 1, It is characterized in that The reinforcement learning algorithm includes a learner module and an actor module; the reinforcement learning algorithm is used to generalize the fusion features and the entity actions in different scenarios to obtain a multi-agent model, including: During the training process, the fused features are input into the learner module for synchronous training to update its own parameters, the fused features and the entity actions are input into the actor module for action decoupling, and a hypothetical scenario is randomly selected for training to update the model parameters. When the reward value or the winning rate reaches a preset value, the next hypothetical scenario is switched to continue training to obtain the multi-agent model.

9. A transferable device for multi-agent reinforcement learning, It is characterized in that The device is applied to a multi-agent military exercise simulation system; the device comprises: The entity layer construction module is used to disassemble the common attributes of entities in the multi-agent environment to obtain common entity classes and corresponding entity attributes; further includes: The common attributes of entities including infantry, tanks, fighters, drones, ships and infrastructure are disassembled to obtain: dynamic attributes including position, speed and ammunition quantity, action capabilities including movement, attack and reconnaissance, environmental interaction including terrain adaptability and concealment, and identity identification including unit number and mission role; A simulation layer construction module, used for defining and encoding entity features based on the general entity class and the entity attributes to obtain an entity feature sequence; A self-attention feature module is used to extract and fuse features of the entity feature sequence to obtain fused features; further comprising: extracting dynamic attributes and action capability features of infantry, tanks, fighters, drones and ships; A self-attention action module, used for decoupling the action of the entity based on the fusion features to obtain the entity action including non-directional action and directional action; A reinforcement learning operator module, used for generalizing the fusion features and the entity actions in different scenarios using a reinforcement learning algorithm to obtain a multi-agent model; The migration module is used to deploy the multi-agent model to the corresponding target scenario to transfer knowledge.

10. A transferable device for multi-agent reinforcement learning, It is characterized in that It includes a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of a transferable method for multi-agent reinforcement learning as described in any one of claims 1 to 8.

Images