A method and system for human swarm interaction based on multi-agent reinforcement learning

Abstract

The application relates to the technical field of agent cooperation, and discloses a method and system for human and cluster interaction based on multi-agent reinforcement learning. The method comprises the following steps: constructing an anti-strategy model of an unmanned mobile platform cluster based on a multi-agent reinforcement learning algorithm; processing the states of the two parties in confrontation by using a state threat evaluation module based on an attention mechanism, and outputting a threat evaluation coefficient of an enemy target; jointly training the state threat evaluation module and the anti-strategy model based on an observation space and the threat evaluation coefficient; fusing a human threat evaluation coefficient and the threat evaluation coefficient to obtain a mixed threat evaluation coefficient; combining the mixed threat evaluation coefficient with an observation state of an unmanned mobile platform, and inputting the mixed threat evaluation coefficient into the trained anti-strategy model to generate and execute cooperative actions for guiding the unmanned mobile platform cluster to carry out confrontation. The application improves the cooperative combat capability and task completion efficiency of the cluster, and can better cope with a changing battlefield environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of agent cooperation, and in particular to a method and system for human and swarm interaction based on multi-agent reinforcement learning. BACKGROUND

[0002] Multi-agent reinforcement learning (MARL) is an important branch of reinforcement learning, which aims to complete tasks through cooperation or competition between multiple agents in a shared environment. In recent years, many MARL algorithms (such as MADDPG algorithm, MAPPO algorithm, VDN algorithm, QMIX algorithm) have made significant progress, thereby promoting research in agent cooperation, strategic optimization and information sharing. However, when facing large-scale and dynamic environments, MARL algorithms still face problems such as poor convergence, difficulty in coordination between agents, and incomplete information. How to improve the cooperation efficiency between agents, optimize the decision-making process and solve these challenges is still the core problem of current multi-agent system research.

[0003] Attention mechanism is an important deep learning method that can help the model focus on the most relevant parts by weighting input information, thereby improving the efficiency of handling complex tasks. In multi-agent reinforcement learning (MARL), attention mechanisms are used to optimize the decision-making process and cooperation strategies of agents, especially in multi-task environments, which can help agents effectively handle information sharing and cooperation problems. Research shows that attention mechanisms not only accelerate the learning of individual agents, but also improve coordination and cooperation in multi-agent systems, making the system more suitable for adaptability and flexibility in dynamic environments.

[0004] Human-machine collaboration is an important research direction in the field of artificial intelligence in recent years. It aims to combine human decision-making ability with the computational advantages of machines to improve the overall performance of the system. In multi-agent reinforcement learning, humans can help agents learn and improve decision-making by providing strategic guidance, adjusting reward mechanisms, or correcting errors. Research shows that human-machine collaboration can compensate for the shortcomings of algorithms and optimize the behavior of agents, especially in the face of uncertainty and complex environments. Through effective interaction, the system can not only improve learning efficiency, but also accelerate the ability to cope with complex tasks and dynamic environments. SUMMARY

[0005] To solve the above technical problems, the present application provides a method and system for human and swarm interaction based on multi-agent reinforcement learning.

[0006] To solve the above technical problems, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a method for human and swarm interaction based on multi-agent reinforcement learning, comprising:

[0008] An adversarial strategy model of the unmanned mobile platform cluster is constructed based on a multi-agent reinforcement learning algorithm, and an observation space and an action space of the adversarial task of the unmanned mobile platform cluster are defined;

[0009] A state threat assessment module based on an attention mechanism is used to process the states of the two opposing parties, and output threat assessment coefficients for each enemy target;

[0010] The state threat assessment module and the adversarial strategy model are jointly trained based on the observation space and the threat assessment coefficients;

[0011] An interactive module with a visual interactive interface is provided for a user to observe the adversarial situation and input human threat assessment coefficients;

[0012] The human threat assessment coefficients are fused with the threat assessment coefficients to obtain mixed threat assessment coefficients;

[0013] The mixed threat assessment coefficients are combined with the observation state of the unmanned mobile platform, and input into the trained adversarial strategy model to generate and execute cooperative actions for guiding the unmanned mobile platform cluster to carry out the confrontation.

[0014] In one embodiment, the definition of the observation space and the action space of the adversarial task of the unmanned mobile platform cluster specifically includes:

[0015] The observation space of the unmanned mobile platform includes the state of the current unmanned mobile platform, the relative state of other unmanned mobile platforms in the friendly cluster, and the relative state of each unmanned mobile platform in the enemy cluster;

[0016] The state of the current unmanned mobile platform includes the position, speed and life value of the unmanned mobile platform in the map; the life value of the unmanned mobile platform refers to the health state of the unmanned mobile platform;

[0017] The relative state of other unmanned mobile platforms in the friendly cluster includes the relative position, relative distance, relative speed and life value of the other unmanned mobile platforms in the friendly cluster relative to the current unmanned mobile platform;

[0018] The relative state of other unmanned mobile platforms in the enemy cluster includes the relative position, relative distance, relative speed, relative attack angle, relative escape angle, speed direction angle and life value of the unmanned mobile platforms in the enemy cluster relative to the current unmanned mobile platform;

[0019] The action space includes various control instructions for controlling the unmanned mobile platform to maneuver.

[0020] In one of the embodiments, the state threat assessment module based on the attention mechanism processes the states of the two opposing parties, and outputs threat assessment coefficients for each enemy target, specifically including:

[0021] The states of each unmanned mobile platform in the own cluster are aggregated by using the self-attention mechanism to obtain a mixed state vector of the own cluster; the mixed state vector of the own cluster is taken as a query vector, the state set of each unmanned mobile platform in the enemy cluster is taken as a key vector and a value vector, the mixed state vector of the enemy cluster is calculated by using the cross-attention mechanism, and threat assessment coefficients for each unmanned mobile platform in the enemy cluster are generated by using the distribution of the attention weight.

[0022] In one of the embodiments, the states of each unmanned mobile platform in the own cluster are aggregated by using the self-attention mechanism to obtain a mixed state vector of the own cluster, specifically including:

[0023] The state of each unmanned mobile platform in the own cluster includes a three-dimensional position coordinate, a three-dimensional velocity vector and a life value, and each state of each unmanned mobile platform in the own cluster is composed into an individual state vector; all individual state vectors of the unmanned mobile platforms in the own cluster are taken as inputs of the self-attention mechanism, and the mixed state vector of the own cluster is outputted by using the aggregation of the self-attention mechanism.

[0024] In one of the embodiments, the mixed state vector of the own cluster is taken as a query vector, the state set of each unmanned mobile platform in the enemy cluster is taken as a key vector and a value vector, the mixed state vector of the enemy cluster is calculated by using the cross-attention mechanism, and threat assessment coefficients for each unmanned mobile platform in the enemy cluster are generated by using the distribution of the attention weight, specifically including:

[0025] The individual state vector of the unmanned mobile platform in the enemy cluster is composed of a three-dimensional position coordinate, a three-dimensional velocity vector and a life value; the mixed state vector of the own cluster is taken as a query vector Q, and the set of individual state vectors of the unmanned mobile platforms in the enemy cluster is taken as a key vector K and a value vector V:

[0026] ;

[0027] The mixed state vector of the enemy cluster is represented by The scaling factor is used to prevent the result of the dot product from being too large, thereby causing the problem of softmax gradient disappearance; wherein The attention coefficient is used to describe the importance of each unmanned mobile platform in the enemy cluster relative to the current mixed state vector of the own cluster, which can be explained as a threat assessment coefficient;

[0028] It can calculate the enemy cluster mixed state vector, where the proportion of each enemy individual's information in the enemy mixed state vector varies depending on the correlation between the enemy individual's state and the friendly mixed state.

[0029] Through the above steps, the state threat assessment module can allocate attention to enemy individuals, giving more attention to enemies that pose a higher threat or are easier to destroy, and prioritizing attacks on these enemies.

[0030] In one embodiment, the process of fusing the human threat assessment coefficient with the threat assessment coefficient output by the state threat assessment module to obtain a hybrid threat assessment coefficient specifically includes:

[0031] ;

[0032] For the mixed threat assessment coefficient, This is the threat assessment coefficient. For assessing human threats, These are the weight parameters.

[0033] In one embodiment, the joint training of the state threat assessment module and the adversarial strategy model based on the observation space and threat assessment coefficients specifically includes:

[0034] The adversarial strategy model includes a value network and a policy network. The value network is used to evaluate the value of the current state, and the policy network is used to select the actions to be performed by each unmanned mobile platform.

[0035] The design includes reward items for advantageous angles, attack range, successful destruction, and mission completion. A threat assessment coefficient is introduced to assign weights to each reward item. The final reward function is obtained by weighted summation, enabling the unmanned mobile platform to prioritize enemy targets with higher threats when making decisions.

[0036] The multi-agent reinforcement learning algorithm adopts a training paradigm of centralized training and distributed execution to train an adversarial policy model. The observation space of the unmanned mobile platform is used as the input of the policy network, and the policy network outputs the current action of the unmanned mobile platform and applies it to the environment. The observation spaces of all unmanned mobile platforms are concatenated into a global state space, which is used as the input of the value network. The value network outputs the value of the current global state space to evaluate the action output by the policy network.

[0037] In one embodiment, when the adversarial task is an air combat task, the unmanned mobile platform is a drone, and the corresponding reward function includes:

[0038] Advantage angle reward items for: ; and These are the attack angle and escape angle of the enemy's j-th drone relative to the ith friendly drone;

[0039] Attack range bonus: Regardless of attack distance, when an enemy drone appears within the attack angle range of your drone, you will receive a bonus of a set value.

[0040] Successful Destruction Reward: When an enemy drone is within the attack angle and range of your drone, and your drone successfully destroys the enemy drone, you will receive a reward of a set value.

[0041] Mission completion reward: Receive a set reward value when all enemy drones are destroyed;

[0042] The threat assessment coefficient corresponding to each reward item is used as the weight for each reward item, with the weight of the task completion reward item being 1. The weighted sum of each reward item is then used to obtain the final reward function.

[0043] In one embodiment, the interactive module providing a visual interactive interface, for users to observe the adversarial situation and input human threat assessment coefficients, specifically includes:

[0044] Through a visual interactive interface, users can at least observe the relative positions of the opposing sides and the assessment results of the state threat assessment module; the visual interactive interface is connected to the backend algorithm via TCP / IP communication protocol to receive the human threat assessment coefficients input by the user.

[0045] In a second aspect, the present invention provides a computer system including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any embodiment of the first aspect.

[0046] Compared with the prior art, the beneficial technical effects of the present invention are:

[0047] Optimize swarm combat strategies: By using the MAPPO algorithm, unmanned mobile platforms can learn the optimal strategy based on the requirements of the environment and the combat mission, thereby improving the swarm's collaborative combat capabilities and mission completion efficiency.

[0048] Improve threat assessment accuracy: Introduce a state threat assessment module based on an attention mechanism, which can accurately assess the threat level of the enemy based on the state information of both sides, thereby enabling the cluster to better cope with changing battlefield environments and improve its combat performance.

[0049] Enhancing Human-Cluster Collaboration: By designing a human-cluster interaction module, users can observe the adversarial situation through a visual interface and receive threat assessments based on prior knowledge. This interaction mechanism makes the algorithm more flexible and allows it to adjust strategies based on user input, thereby further improving the effectiveness of cluster adversarial operations in complex environments. Attached Figure Description

[0050] Figure 1 This is a flowchart of the method in an embodiment of the present invention. Detailed Implementation

[0051] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0052] like Figure 1 As shown, this invention provides a method for human-cluster interaction based on multi-agent reinforcement learning, comprising the following steps:

[0053] S1, Construct an adversarial strategy model for an unmanned mobile platform cluster based on a multi-agent reinforcement learning algorithm, and define the observation space and action space of the adversarial task of the unmanned mobile platform cluster.

[0054] S2 utilizes an attention-based state threat assessment module to process the states of both adversaries and outputs threat assessment coefficients for each enemy target.

[0055] S3, based on the observation space and threat assessment coefficients, jointly trains the state threat assessment module and the adversarial strategy model;

[0056] S4 provides an interactive module with a visual interface, allowing users to observe the adversarial situation and input human threat assessment coefficients.

[0057] S5 combines the human threat assessment coefficient with the threat assessment coefficient to obtain the hybrid threat assessment coefficient;

[0058] S6. The hybrid threat assessment coefficient is combined with the observation status of the unmanned mobile platform and input into the trained adversarial strategy model to generate and execute cooperative actions that guide the unmanned mobile platform cluster to conduct adversarial operations.

[0059] The main framework of this invention is based on the Multi-Agent Reinforcement Learning (MAPPO) algorithm. Using MAPPO as the foundational algorithm, the algorithm takes the observed state of the unmanned mobile platform as input and the actions of the unmanned mobile platform as output, training it to obtain the optimal strategy for the unmanned mobile platform swarm in adversarial tasks, thereby fulfilling the task requirements of swarm adversarial warfare.

[0060] Based on this, a state threat assessment module based on an attention mechanism is designed. The state information of both the enemy and ourselves is used as input, and the threat assessment of the enemy is used as output. A neural network model is trained to judge the threat level of individual enemies based on the current enemy and ourselves situation, so as to improve the performance of cluster confrontation tasks.

[0061] Finally, an interaction module was designed, including an algorithm interaction interface and a visual interaction interface. Users can observe the adversarial situation through the visual interface and generate a user threat assessment based on their prior knowledge. This information is then transmitted to the algorithm through the visual interaction interface, so that the algorithm's output action can be reasonably changed according to the user's input, thereby further improving the performance of the cluster adversarial task.

[0062] In one embodiment, step S1 specifically includes:

[0063] Define the observation space and action space for multi-unmanned mobile platform swarm combat missions. The observation space includes the current position, speed, and other status information of the unmanned mobile platforms on the map, as well as the status information of teammates and enemy targets; the unmanned mobile platforms can be drones, robots, etc. The action space contains a series of control commands to control the unmanned mobile platforms and enemy targets to maneuver, such as throttle, aileron, and rudder control commands in multi-drone combat missions.

[0064] Initialize the value network and policy network. Both are randomly initialized networks. The value network evaluates the value of the current state, while the policy network selects the action to be performed for each unmanned mobile platform. The value network is constructed from four fully connected layers, with each hidden layer having a width of 256. Similarly, the policy network contains four fully connected layers, with each hidden layer having a width of 256.

[0065] In one embodiment, step S2 specifically includes:

[0066] Initialize the attention network and use self-attention and cross-attention mechanisms for threat assessment.

[0067] The self-attention mechanism is used to aggregate information from the user's own cluster. In this module, the state of each unmanned mobile platform in the user's cluster includes three-dimensional position coordinates. Three-dimensional velocity vector Together with health points, this information forms an individual state vector. All individual state vectors in the cluster are combined into a two-dimensional vector, which serves as the input to the self-attention mechanism. Specifically, the query vector Q, key vector K, and value vector V of the self-attention mechanism are all this two-dimensional vector. After aggregation by the self-attention mechanism, a mixed state vector of the cluster is output, containing partial information from each individual in the cluster to represent the average state of the cluster. The health points of an unmanned mobile platform refer to the current health status of the unmanned mobile platform (e.g., a drone), with an initial health point typically set to 100. When the unmanned mobile platform enters the pre-defined attack range of an enemy unit, this health point will, with a certain probability, become 0, meaning the unmanned mobile platform is destroyed by the enemy and loses its ability to perform missions.

[0068] In the cross-attention mechanism section, the role of the cross-attention mechanism is to describe the relationship between the mixed state vector of the friendly cluster and the state vector of each individual enemy. Similarly, the state vector of an individual enemy also contains three-dimensional position coordinates. Three-dimensional velocity vector And health information. The friendly cluster's mixed state vector is used as the query vector Q, and the enemy's individual state vector set is used as the key vector K and value vector V, calculated using the following formula:

[0069] ;

[0070] It is possible to calculate an enemy mixed state vector, where the proportion of each enemy individual's information in the enemy mixed state vector varies depending on the correlation between the enemy individual's state and the friendly mixed state. It is used to describe this correlation, that is, the degree of importance that one's own group attaches to a certain enemy individual.

[0071] Through the above steps, the state threat assessment module can allocate attention to enemy individuals, giving more attention to enemies that pose a higher threat or are easier to destroy, and prioritizing attacks on these enemies.

[0072] The state threat assessment module needs to be jointly trained with the adversarial strategy model. During the training phase, the state threat assessment module needs to add the assessment results of the adversarial individual to the information of each enemy individual observed by the unmanned mobile platform. Throughout this process, not only are the model parameters of the policy network and the value network iteratively updated, but the parameters of the self-attention mechanism and the cross-attention mechanism are also updated. In the end, the optimal policy can be obtained while also obtaining a relatively accurate state threat assessment model.

[0073] In one embodiment, step S3 specifically includes:

[0074] The multi-agent reinforcement learning algorithm combines state information with the threat assessment coefficients output by the state threat assessment module, trains them together with the state threat assessment module, and guides the autonomous decision-making and actions of the cluster individuals through the policy network.

[0075] Design a state observation space for the unmanned mobile platform. The observation space comprises three parts: self-information, teammate information, and enemy information. Self-information includes the platform's three-dimensional position coordinates. Three-dimensional velocity vector And health points.

[0076] Teammate information includes the three-dimensional relative position vector of each teammate. Relative distance, three-dimensional relative velocity vector And health points; among them, The teammate's three-dimensional position coordinates, The three-dimensional velocity vector of the teammate.

[0077] Enemy information includes the three-dimensional relative position vector of each enemy. Relative distance, three-dimensional relative velocity vector The relative attack angle, relative escape angle, speed direction angle, and health points. The enemy's three-dimensional position coordinates. This represents the enemy's three-dimensional velocity vector.

[0078] The observation space of each unmanned mobile platform is composed of the above three parts of information and serves as the input to the policy network. The global state space is composed of the observation spaces of each unmanned mobile platform. The observation spaces of each unmanned mobile platform are combined into a large state space as the global state space and serve as the input to the value network.

[0079] Designing the motion space for an unmanned mobile platform. In a preferred embodiment, swarm air combat is selected as the background, and a dynamic model of a real fixed-wing fighter jet is added. Therefore, ailerons, elevators, rudders, and throttles are selected as the motion space. However, the swarm combat scenarios applicable to this method are not limited to swarm air combat; therefore, the motion space can be designed as the underlying motions of an unmanned mobile platform in any swarm combat scenario.

[0080] Designing a reasonable reward function based on threat assessment information. Designing a reasonable reward function requires considering multiple factors. This method considers factors such as angle, distance, success / failure of the confrontation, and task completion when designing the reward function. The advantageous angle reward encourages the agent to avoid the enemy's attack range and attack the enemy. For example, in an air combat scenario, choosing to attack from behind the enemy, the formula for the reward function is:

[0081] .

[0082] in, and These are the attack angle and escape angle of the enemy's j-th drone relative to the ith friendly drone. Another angle-related reward is the attack range reward, which, regardless of attack distance, awards a certain value when the enemy appears within the friendly drone's attack angle range. This reward encourages friendly drones to move towards the enemy even at greater distances. The successful destruction reward considers both angle and distance. When the enemy's relative position simultaneously meets the friendly drone's attack angle and distance conditions, and the drone successfully destroys the enemy, a relatively large event reward is awarded, indicating the event's importance. Similarly, if the drone meets the enemy's attack angle and distance conditions, a relatively large event penalty is awarded. Finally, when the mission is successfully completed—that is, the enemy is completely eliminated—an even larger event reward is awarded, indicating that this event represents the desired outcome of the optimal strategy. Simultaneously, the reward function incorporates corresponding threat information as a weighting coefficient for the reward value to strengthen the impact of threat information on the drone's action decisions.

[0083] The policy network outputs action information based on state and threat assessment information. The unmanned mobile platform executes actions that affect the environmental state, and the environment provides corresponding rewards based on the current state and the actions of the unmanned mobile platform. In this way, the unmanned mobile platform continuously interacts with the environment to obtain... Training data in the form of, where These represent the current state, current action, current reward, and next moment's state, respectively.

[0084] Using the collected training data, the multi-agent reinforcement learning algorithm continuously updates and iterates the policy of the unmanned mobile platform. During the algorithm training process, a centralized training distributed execution (CTDE) paradigm is used to train an actor neural network (i.e., the policy network) and a centralized critic neural network (i.e., the value network). For all individual unmanned mobile platforms in the cluster, the policy network receives the observation state information of each individual platform, outputs the current action of each platform, and applies it to the environment, causing a corresponding change in the environment state. The centralized value network receives the global state information of the environment, or it can be an aggregation of the observation information of all individual unmanned mobile platforms, and outputs the value of the current global state information. This value is used to evaluate the quality of the unmanned mobile platform actions output by the policy network, guiding the policy network's updates. The CTDE training paradigm indicates that both the policy network and the value network need to be used and updated during the training phase, while only the trained policy network is used during the execution phase to generate actions for individual unmanned mobile platforms based on the current observation state.

[0085] The value network and policy network are trained at this stage, resulting in an optimal policy network. In the execution stage after the training phase, only the trained policy network is needed. The observation space of the unmanned mobile platform is used as the input to the policy network, and the actions output by the policy network are applied to the environment by the unmanned mobile platform to complete the swarm adversarial task.

[0086] In one embodiment, step S4 specifically includes:

[0087] Develop a user-friendly interactive module that includes a visual interface developed using Python and the OpenGL library. Through this interface, users can observe the relative positions of friendly and enemy forces in a 3D scene, the assessment results from the threat assessment module, and some log information.

[0088] While receiving information, users can manually assign the importance of each enemy entity through this visual interactive interface, i.e., the user's attention allocation to enemy entities. The front-end interface and back-end algorithm communicate via TCP / IP protocol, and the human threat assessment coefficient set by the user can be obtained by the back-end algorithm through the communication protocol.

[0089] In one embodiment, step S5 specifically includes:

[0090] To verify the algorithm's performance, this invention designed a 3D air combat environment suitable for multi-agent reinforcement learning. It not only adds a highly realistic fixed-wing UAV dynamics model but also includes environmental factors such as air resistance and density, maximizing the recreation of realistic air combat scenarios. The environment is developed and built using Python based on the Harfang 3D physics engine and features various models of fixed-wing fighter jets, air-to-air missiles, and surface-to-air missiles, capable of meeting the mission requirements of various swarm combat scenarios.

[0091] The user-defined human threat assessment coefficient is combined with the threat assessment coefficient output by the state threat assessment module:

[0092] ;

[0093] This yields the hybrid threat assessment coefficients, which serve as input to the adversarial strategy model.

[0094] The policy network in the adversarial strategy model outputs action information based on the hybrid threat assessment coefficient, which can more accurately assess the threat level of each enemy individual and help the friendly cluster make more reasonable judgments and actions.

[0095] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0096] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0097] In one embodiment, a computer system is provided, which may be a server. The computer system includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data used in the methods described above. The network interface communicates with external terminals via a network connection. The computer program is executed by the processor to implement the methods described above.

[0098] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0100] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for human-cluster interaction based on multi-agent reinforcement learning, characterized in that, include: An adversarial strategy model for an unmanned mobile platform cluster is constructed based on a multi-agent reinforcement learning algorithm, defining the observation space and action space of the adversarial task for the unmanned mobile platform cluster. The state threat assessment module based on the attention mechanism processes the states of both adversaries and outputs threat assessment coefficients for each enemy target. Specifically, it includes: using a self-attention mechanism to aggregate the states of each unmanned mobile platform in the friendly cluster to obtain a mixed state vector of the friendly cluster; using the mixed state vector of the friendly cluster as the query vector, and the state set of each unmanned mobile platform in the enemy cluster as the key vector and value vector, calculating the mixed state vector of the enemy cluster through a cross-attention mechanism, and generating threat assessment coefficients for each unmanned mobile platform in the enemy cluster through the allocation of attention weights. The state threat assessment module and adversarial strategy model are jointly trained based on the observation space and threat assessment coefficients. Specifically, the adversarial strategy model includes a value network and a policy network. The value network is used to evaluate the value of the current state, and the policy network is used to select actions for each unmanned mobile platform. Advantage angle reward items, attack range reward items, successful destruction reward items, and task completion reward items are designed, and threat assessment coefficients are introduced to assign weights to each reward item. The final reward function is obtained through weighted summation, enabling the unmanned mobile platform to prioritize enemy targets with higher threats when making decisions. The multi-agent reinforcement learning algorithm adopts a training paradigm of centralized training and distributed execution to train the adversarial strategy model. The observation space of the unmanned mobile platform is used as the input to the policy network, and the policy network outputs the current actions of the unmanned mobile platform and applies them to the environment. The observation spaces of all unmanned mobile platforms are concatenated into a global state space, which is used as the input to the value network. The value network outputs the value of the current global state space to evaluate the actions output by the policy network. Provides an interactive module with a visual interface for users to observe the confrontation situation and input human threat assessment coefficients; By fusing the human threat assessment coefficient with the threat assessment coefficient, a hybrid threat assessment coefficient is obtained; The hybrid threat assessment coefficients are combined with the observed state of the unmanned mobile platform and input into the trained adversarial strategy model to generate and execute cooperative actions that guide the unmanned mobile platform cluster in adversarial operations.

2. The method for human-cluster interaction based on multi-agent reinforcement learning according to claim 1, characterized in that, The definition of the observation space and action space for unmanned mobile platform cluster adversarial tasks specifically includes: The observation space of the unmanned mobile platform includes the current state of the unmanned mobile platform, the relative state of other unmanned mobile platforms in the friendly cluster, and the relative state of each unmanned mobile platform in the enemy cluster. The current status of the unmanned mobile platform includes its position on the map, speed, and health points; the health points of the unmanned mobile platform refer to its health status. The relative status of other unmanned mobile platforms in the friendly cluster includes their relative position, relative distance, relative speed, and health points relative to the current unmanned mobile platform. The relative status of other unmanned mobile platforms in the enemy cluster includes the relative position, relative distance, relative speed, relative attack angle, relative escape angle, speed direction angle, and health points of the enemy cluster's unmanned mobile platforms relative to the current unmanned mobile platform; The motion space includes various control commands that control the unmanned mobile platform to perform maneuvers.

3. The method for human-cluster interaction based on multi-agent reinforcement learning according to claim 1, characterized in that, The method of aggregating the states of each unmanned mobile platform in the user cluster using a self-attention mechanism to obtain a hybrid state vector of the user cluster specifically includes: The state of each unmanned mobile platform in the friendly cluster includes three-dimensional position coordinates, three-dimensional velocity vector, and health value. The states of each unmanned mobile platform in the friendly cluster are combined into an individual state vector. All individual state vectors of the unmanned mobile platforms in the friendly cluster are used as input to the self-attention mechanism. After aggregation by the self-attention mechanism, the mixed state vector of the friendly cluster is output.

4. The method for human-cluster interaction based on multi-agent reinforcement learning according to claim 1, characterized in that, The method uses the mixed state vector of the friendly cluster as the query vector and the state sets of each unmanned mobile platform in the enemy cluster as the key and value vectors. Through a cross-attention mechanism, the mixed state vector of the enemy cluster is calculated, and the threat assessment coefficients for each unmanned mobile platform in the enemy cluster are generated by allocating attention weights. Specifically, this includes: The individual state vector of an unmanned mobile platform in the enemy cluster consists of three-dimensional position coordinates, a three-dimensional velocity vector, and health points; the mixed state vector of the friendly cluster is used as the query vector Q, and the set of individual state vectors of unmanned mobile platforms in the enemy cluster is used as the key vector K and the value vector V. ； This represents the mixed state vector of the enemy cluster. Represents the scaling factor; where This represents the threat assessment coefficient.

5. The method for human-cluster interaction based on multi-agent reinforcement learning according to claim 1, characterized in that, The process of fusing the human threat assessment coefficient with the threat assessment coefficient output by the state threat assessment module to obtain a hybrid threat assessment coefficient specifically includes: ； For the mixed threat assessment coefficient, This is the threat assessment coefficient. For assessing human threats, These are the weight parameters.

6. The method for human-cluster interaction based on multi-agent reinforcement learning according to claim 1, characterized in that, When the adversarial mission is an air combat mission, the unmanned mobile platform is a drone, and the corresponding reward function includes: Advantage angle reward items for: ; and These are the attack angle and escape angle of the enemy's j-th drone relative to the ith friendly drone; Attack range bonus: Regardless of attack distance, when an enemy drone appears within the attack angle range of your drone, you will receive a bonus of a set value. Successful Destruction Reward: When an enemy drone is within the attack angle and range of your drone, and your drone successfully destroys the enemy drone, you will receive a reward of a set value. Mission completion reward: Receive a set reward value when all enemy drones are destroyed; The threat assessment coefficient corresponding to each reward item is used as the weight for each reward item, with the weight of the task completion reward item being 1. The weighted sum of each reward item is then used to obtain the final reward function.

7. The method for human-cluster interaction based on multi-agent reinforcement learning according to claim 1, characterized in that, The interactive module, which provides a visual interface for users to observe the adversarial situation and input human threat assessment coefficients, specifically includes: Through a visual interactive interface, users can at least observe the relative positions of the opposing sides and the assessment results of the state threat assessment module; the visual interactive interface is connected to the backend algorithm via TCP / IP communication protocol to receive the human threat assessment coefficients input by the user.

8. A computer system comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images