Multi-agent dynamic defense game method and system based on federal reinforcement learning

Abstract

The application discloses a multi-agent dynamic defense game method and system based on federal reinforcement learning, and relates to the technical field of information security.The application comprises the following steps: modeling a distributed network defense scene as a multi-agent based partially observable Markov environment, defining a seven-tuple environment model comprising an agent set, a global state space, a local observation space, an action space, a state transition function, an observation function and a reward function, and designing a hierarchical mixed reward function; constructing a hierarchical collaborative defense architecture based on federal reinforcement learning, and executing a closed-loop online dynamic defense process based on the defense architecture.The application executes distributed training and real-time decision-making through a local agent layer, performs intra-group model encryption aggregation in an edge layer, and completes global meta-strategy generation and dynamic role allocation in a central layer.The architecture reduces communication overhead, avoids single-point failures and effectively protects the privacy of original data of each node, and is conducive to efficient adaptive strategy learning and dynamic game collaboration.

Description

Technical Field

[0001] This invention belongs to the field of information security technology, and in particular relates to a multi-agent dynamic defense game method and system based on federated reinforcement learning. Background Technology

[0002] With the rapid development of network technology, network attacks have become increasingly diversified, covert, and collaborative, posing a severe challenge to security defense in distributed network environments. To overcome the passive situation of being easy to attack but difficult to defend, dynamic defense technology has emerged. Its core idea is to actively and randomly change system configurations or network states to increase the difficulty and cost of attack for attackers. Reinforcement learning, due to its environmental adaptability and policy self-optimization characteristics, is widely used in dynamic defense research, aiming to enable defense nodes to learn optimal strategies through interaction with the attack environment. However, as defense scenarios extend to large-scale distributed networks such as large enterprise networks, cloud data centers, and edge computing networks, existing technologies are gradually revealing limitations at both the architectural and mechanistic levels.

[0003] On the one hand, existing collaborative defense architectures face structural defects when dealing with large-scale distributed scenarios. Current mainstream methods mostly employ centralized training architectures, which, while theoretically achieving globally optimal collaboration, suffer from the following problems in practical deployment: First, frequent communication between massive numbers of defense nodes and the central control defense node leads to huge communication overhead and latency, becoming a bottleneck for system performance. Second, the existence of the central defense node introduces the risk of single point of failure; once compromised or rendered ineffective, the entire defense system will collapse. Third, each defense node needs to upload its local observation data or model gradients to the central server, a process fraught with the risk of data leakage, especially in scenarios involving sensitive network topologies or user privacy, where centralized training struggles to meet privacy compliance requirements. Therefore, although distributed training paradigms such as federated learning have demonstrated the potential for decentralized collaboration in some areas, how to combine them with the real-time requirements of dynamic defense remains a pressing challenge.

[0004] On the other hand, existing technologies suffer from a lack of mechanistic understanding in modeling attack and defense: most reinforcement learning-driven defense strategies remain at the passive adaptation level of "stimulus-response," lacking explicit modeling of the interaction process between the attacking and defending strategies. Attackers' behavior is not random noise, but rather rational decision-making with clear objectives; defenders' strategy choices should also be based on predictions of possible attacker strategies. Most methods focus only on local policy learning or simple collaboration of agents, lacking explicit modeling of the attack-defense game process. This makes it difficult to achieve dynamic matching between defense and attack strategies, resulting in insufficient adaptability and collaborative efficiency of the defense system. It is unable to cope with complex and ever-changing distributed network attack scenarios; that is, defense nodes cannot dynamically adjust their functions according to changes in attacker strategies, nor can they switch between different defense roles such as monitoring, blocking, trapping, and handling as needed. While existing technologies also incorporate game theory frameworks to model the attack-defense process, they often face theoretical bottlenecks such as state space explosion and insufficient modeling capabilities for incomplete information, making it difficult to address information security issues in real-world network environments.

[0005] Therefore, how to enable multiple defense nodes to achieve autonomous adaptation and efficient collaboration in a partially observable and dynamically adversarial network environment is the technical problem that this invention aims to solve. Summary of the Invention

[0006] The purpose of this invention is to provide a multi-agent dynamic defense game method and system based on federated reinforcement learning. This method models the distributed network defense scenario as a partially observable Markov environment based on multiple agents and designs a hierarchical hybrid reward function. Based on federated reinforcement learning, a three-layer architecture is constructed: a local agent layer, an edge federated aggregation layer, and a central coordination layer. The local agent layer performs local training and experience collection based on the SAC algorithm. The edge federated aggregation layer periodically aggregates models of defense nodes within the group to generate edge policy prototypes. The central coordination layer selects and merges the edge policy prototypes to generate a global meta-policy and performs meta-learning fine-tuning. Based on this, a meta-game model is constructed and dynamic roles are dynamically assigned. This invention solves the limitations of existing multi-agent reinforcement learning-based dynamic defense methods in terms of communication overhead, single points of failure and data leakage in centralized training architectures, and lack of dynamic game modeling.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] As the first aspect provided by this invention, this invention provides a multi-agent dynamic defense game method and system based on federated reinforcement learning, comprising the following steps:

[0009] The distributed network defense scenario is modeled as a partially observable Markov environment based on multiple agents. A seven-tuple environment model is defined, which includes a set of agents, a global state space, a local observation space, an action space, a state transition function, an observation function, and a reward function. A hierarchical hybrid reward function is designed.

[0010] Construct a hierarchical collaborative defense architecture based on federated reinforcement learning. The defense architecture includes:

[0011] The local agent layer consists of agents that use the Soft Actor-Critic (SAC) algorithm to train local policies and collect experience based on local observations.

[0012] The edge federated aggregation layer periodically aggregates the local model parameters of agents within the group to generate edge policy prototypes.

[0013] The central coordination layer evaluates, selects, and integrates various peripheral strategy prototypes to generate a global meta-strategy and performs meta-learning fine-tuning. Based on the global meta-strategy, it constructs an offensive and defensive meta-game model, solves the hybrid strategy Nash equilibrium, and performs dynamic role allocation based on the hybrid strategy Nash equilibrium.

[0014] Based on the defense architecture, a closed-loop online dynamic defense process is executed. The closed-loop online dynamic defense process includes local decision-making and experience collection, continuous local learning and policy updates, edge-layer federated aggregation and policy fusion, and central-layer global coordination and role allocation. This process enables multiple defense nodes to learn adaptive policies and conduct efficient collaborative defense in a dynamic adversarial environment.

[0015] Furthermore, the hierarchical hybrid reward function is obtained by weighted summation of local reward, collaborative reward, and global reward, wherein:

[0016] Local rewards, used to evaluate the direct effect of the agent's own defensive actions, are calculated by weighting the interception success rate, resource consumption rate, and false alarm rate.

[0017] Collaboration rewards, used to encourage collaborative behavior among agents, are calculated by weighting the similarity of actions and the number of policy conflicts among agents.

[0018] The global reward, which reflects the overall security status of the network, is calculated by weighting the change in the global security score and the degree of attack spread.

[0019] Instant rewards are calculated using a hierarchical hybrid reward function.

[0020] Furthermore, the agent maintains a local experience replay buffer and periodically samples experience data from the buffer, updating the action value network and policy network by minimizing the loss function; the method for local policy training based on the Soft Actor-Critic (SAC) algorithm includes:

[0021] Perform network initialization: initialize the policy network parameters and action value network parameters, and obtain the corresponding target network parameters;

[0022] Randomly sample batches of experience data from the local experience replay buffer;

[0023] For each sampling experience, the target Q value is calculated using the minimum strategy in the target network to suppress overestimation;

[0024] Update the action value network and policy network by minimizing the loss function;

[0025] The target network is soft-updated to ensure training stability; the loss function of the policy network combines Q-value estimation and policy entropy, and the temperature parameter can be automatically adjusted according to the policy entropy to balance exploration and exploitation.

[0026] Furthermore, the method for performing periodic federated aggregation in the edge federated aggregation layer includes:

[0027] Multiple agents are divided into edge groups, and each edge group is assigned an edge aggregation defense node;

[0028] The edge aggregation defense node requests and collects policy network parameters that have been homomorphically encrypted from the agents within the group;

[0029] Based on the amount of empirical data, defense effectiveness, and model quality evaluation, the aggregate weight of each agent is calculated.

[0030] An edge policy prototype is generated using a weighted linear fusion method, and the policy network parameters are pruned to prevent parameter explosion.

[0031] The edge policy prototype is distributed to agents within the group, and the agents update their local parameters through policy fusion.

[0032] Furthermore, the method for generating the global meta-policy includes:

[0033] The central coordination layer maintains a global verification environment, periodically collects edge policy prototypes from each edge aggregation defense node, and deploys the edge policy prototypes in the global verification environment;

[0034] In a global verification environment, the defense performance of edge policy prototypes is evaluated and screened based on a global security scoring function. The screened edge policy prototypes are then weighted and linearly fused to generate an initial global meta-policy.

[0035] The updated global meta-policy is obtained by fine-tuning the meta-learning based on the meta-learning framework. The method of fine-tuning the meta-learning based on the meta-learning framework is as follows: the defense environment of each edge group is regarded as a meta-task, and the global meta-policy is made capable of quickly adapting to different edge environments through inner loop task adaptation and outer loop global policy update. The meta-learning framework is the Model-Agnostic Meta-Learning framework.

[0036] Furthermore, the central coordination layer is also used to construct a meta-game model and execute dynamic role allocation. The methods for constructing the meta-game model and executing dynamic role allocation include:

[0037] Construct and dynamically update the attack and defense benefit matrix;

[0038] The offensive and defensive game is modeled as a two-player zero-sum game, and the mixed-policy Nash equilibrium is solved to obtain the optimal policy distribution of the defender.

[0039] Based on hardware performance adaptability, location adaptability, and historical performance scores, the role adaptability of each agent to the four defensive roles of monitoring, blocking, trapping, and handling is calculated.

[0040] Based on the hybrid strategy Nash equilibrium and the role fit of each agent, the role allocation optimization problem is solved to obtain the optimal role allocation scheme. The role fit is calculated based on hardware performance fit, location fit, and historical performance score. The constraints of the role allocation optimization problem include that each agent is assigned only one role and the upper limit of the number of each role.

[0041] Furthermore, the closed-loop online dynamic defense process includes:

[0042] Local decision-making and experience collection: The agent generates and executes defensive actions based on the current policy network and role preference vector, obtains immediate rewards consisting of local rewards, cooperative rewards and global rewards, and stores the experience in the local experience replay buffer;

[0043] Continuous local learning and policy update: The agent periodically samples experience data from the buffer and performs local training based on the Soft Actor-Critic algorithm to update the policy;

[0044] Edge layer federated aggregation and policy fusion: After each preset round of local training is completed, the edge aggregation defense node aggregates the policy parameters of the agents in the group to generate an edge policy prototype and distributes it. After receiving the edge policy prototype, the agents synchronize their local parameters.

[0045] Central-level global coordination and role allocation: After completing each preset round of edge federation aggregation, the central coordination defense node aggregates the prototypes of each edge strategy to generate and fine-tune the global meta-strategy. At the same time, it constructs the meta-game model and dynamically allocates roles, and distributes the updated global meta-strategy and role allocation scheme to each agent. The agents update their local parameters and switch roles.

[0046] As a second aspect of the present invention, the present invention provides a multi-agent dynamic defense game system based on federated reinforcement learning, the system being used to implement the method described in the first aspect, the system comprising:

[0047] The environment modeling module is used to model distributed network defense scenarios as a partially observable Markov environment based on multi-agent systems and to design a hierarchical hybrid reward function.

[0048] The local agent layer consists of heterogeneous defense nodes distributed in the network. The defense nodes act as agents and perform local policy training and experience collection through the Soft Actor-Critic (SAC) algorithm.

[0049] The edge federated aggregation layer, which consists of multiple edge aggregation defense nodes, is responsible for periodically federating the model parameters of the agents within the group to generate edge policy prototypes.

[0050] The central coordination layer, which consists of central coordination defense nodes, is responsible for integrating various edge strategy prototypes, generating global meta-policies and performing meta-learning fine-tuning, while constructing an offensive and defensive meta-game model and executing dynamic role allocation.

[0051] The online decision-making module is used to execute a closed-loop online defense process, enabling multiple defense nodes to learn adaptive strategies and cooperate in defense in a dynamic adversarial environment.

[0052] Furthermore, the action space of the agent in the local agent layer includes: access control actions, traffic management actions, detection and response actions, deception deployment actions, and system configuration actions. When generating defense actions, the policy network output is weighted by the role preference vector to ensure that the actions conform to the current role function.

[0053] Furthermore, the central coordinating layer also includes:

[0054] The global verification environment module is used to test and evaluate the defense performance of edge strategy prototypes.

[0055] The game solving module is used to model the attack-defense meta-game model as a two-player zero-sum game and solve the mixed-strategy Nash equilibrium.

[0056] The role allocation module is used to solve for the optimal role allocation scheme based on role suitability and hybrid strategy Nash equilibrium.

[0057] The present invention has the following beneficial effects:

[0058] This invention models a distributed network defense scenario as a partially observable Markov environment based on multi-agent systems and designs a hierarchical hybrid reward function. It constructs a three-layer architecture based on federated reinforcement learning: a local agent layer, an edge federated aggregation layer, and a central coordination layer. The local agent layer performs local training and experience collection based on the SAC algorithm; the edge federated aggregation layer periodically aggregates models of defense nodes within a group to generate edge policy prototypes; the central coordination layer selects and merges these edge policy prototypes to generate a global meta-policy and performs meta-learning fine-tuning. Based on this, a meta-game model is constructed and dynamic role allocation is performed. Ultimately, a distributed multi-agent defense system capable of adaptive policy learning and dynamic game collaboration is built, enabling autonomous adaptation and efficient collaborative proactive defense capabilities among multiple defense nodes in a partially observable, dynamically adversarial network environment.

[0059] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a flowchart of the multi-agent dynamic defense game method based on federated reinforcement learning of the present invention.

[0062] Figure 2 This is a block diagram of the overall scheme of the multi-agent dynamic defense game method based on federated reinforcement learning of the present invention;

[0063] Figure 3 This is a block diagram of the multi-agent dynamic defense game system based on federated reinforcement learning of the present invention;

[0064] Figure 4 This is a schematic diagram illustrating the deployment of the large-scale enterprise cloud network environment solution of the present invention. Detailed Implementation

[0065] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0066] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0067] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0068] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0069] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0070] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0071] Example 1:

[0072] Please see Figure 1-2As shown, as the first aspect provided by this invention, this invention is a multi-agent dynamic defense game method based on federated reinforcement learning, comprising the following steps:

[0073] The distributed network defense scenario is modeled as a partially observable Markov environment based on multiple agents. A seven-tuple environment model is defined, which includes a set of agents, a global state space, a local observation space, an action space, a state transition function, an observation function, and a reward function. A hierarchical hybrid reward function is designed.

[0074] Construct a hierarchical collaborative defense architecture based on federated reinforcement learning. The defense architecture includes:

[0075] The local agent layer consists of agents that use the Soft Actor-Critic (SAC) algorithm to train local policies and collect experience based on local observations.

[0076] The edge federated aggregation layer periodically aggregates the local model parameters of agents within the group to generate edge policy prototypes.

[0077] The central coordination layer evaluates, selects, and integrates various peripheral strategy prototypes to generate a global meta-strategy and performs meta-learning fine-tuning. Based on the global meta-strategy, it constructs an offensive and defensive meta-game model, solves the hybrid strategy Nash equilibrium, and performs dynamic role allocation based on the hybrid strategy Nash equilibrium.

[0078] Based on the defense architecture, a closed-loop online dynamic defense process is executed. The closed-loop online dynamic defense process includes local decision-making and experience collection, continuous local learning and policy updates, edge-layer federated aggregation and policy fusion, and central-layer global coordination and role allocation. This process enables multiple defense nodes to learn adaptive policies and conduct efficient collaborative defense in a dynamic adversarial environment.

[0079] As an embodiment of the present invention, preferably, the hierarchical hybrid reward function is obtained by weighted summation of local reward, collaborative reward, and global reward, wherein:

[0080] Local rewards, used to evaluate the direct effect of the agent's own defensive actions, are calculated by weighting the interception success rate, resource consumption rate, and false alarm rate.

[0081] Collaboration rewards, used to encourage collaborative behavior among agents, are calculated by weighting the similarity of actions and the number of policy conflicts among agents.

[0082] The global reward, which reflects the overall security status of the network, is calculated by weighting the change in the global security score and the degree of attack spread.

[0083] Instant rewards are calculated using a hierarchical hybrid reward function.

[0084] As an embodiment of the present invention, preferably, based on the global meta-policy generation at the central coordination layer, a meta-game model is further constructed and dynamic role allocation is executed to achieve explicit modeling and intelligent collaborative scheduling of the attack and defense process. This mechanism establishes an attack and defense payoff matrix, models the attack and defense game as a two-player zero-sum game, solves its mixed-policy Nash equilibrium, and performs dynamic role allocation. The method for constructing the meta-game model and executing dynamic role allocation includes:

[0085] Construct and dynamically update the attack and defense benefit matrix;

[0086] The offensive and defensive game is modeled as a two-player zero-sum game, and the mixed-policy Nash equilibrium is solved to obtain the optimal policy distribution of the defender.

[0087] Based on hardware performance adaptability, location adaptability, and historical performance scores, the role adaptability of each agent to the four defensive roles of monitoring, blocking, trapping, and handling is calculated.

[0088] Based on the hybrid strategy Nash equilibrium and the role suitability of each agent, the role allocation optimization problem is solved to obtain the optimal role allocation scheme.

[0089] As an embodiment of the present invention, preferably, the closed-loop online dynamic defense process includes:

[0090] Local decision-making and experience collection: The agent generates and executes defensive actions based on the current policy network and role preference vector, obtains immediate rewards consisting of local rewards, cooperative rewards and global rewards, and stores the experience in the local experience replay buffer;

[0091] Continuous local learning and policy update: The agent periodically samples experience data from the buffer and performs local training based on the Soft Actor-Critic algorithm to update the policy;

[0092] Edge layer federated aggregation and policy fusion: After each preset round of local training is completed, the edge aggregation defense node aggregates the policy parameters of the agents in the group to generate an edge policy prototype and distributes it. After receiving the edge policy prototype, the agents synchronize their local parameters.

[0093] Central-level global coordination and role allocation: After completing each preset round of edge federation aggregation, the central coordination defense node aggregates the prototypes of each edge strategy to generate and fine-tune the global meta-strategy. At the same time, it constructs the meta-game model and dynamically allocates roles, and distributes the updated global meta-strategy and role allocation scheme to each agent. The agents update their local parameters and switch roles.

[0094] Example 2:

[0095] As an embodiment of the present invention, preferably based on Embodiment 1, please refer to... Figure 1-2 As shown, defense nodes need to achieve collaborative detection and dynamic response when they can only obtain local network information. The distributed network defense scenario is modeled as a partially observable Markov environment based on multiple agents, which contains N heterogeneous defense nodes, including firewall defense nodes, intrusion detection system defense nodes, endpoint protection agent defense nodes, network traffic analysis defense nodes, and honeypot defense nodes. Each defense node is distributed in different network areas, forming a collaborative defense system.

[0096] As an embodiment of the present invention, preferably, the partially observable Markov environment is defined as a seven-tuple. Including a collection of intelligent agents Global state space Local observation space Action space State transition function Observation function and reward function .

[0097] As an embodiment of the present invention, preferably, a set of intelligent agents. This means that N heterogeneous defense nodes are modeled as a set of agents, where each agent corresponds to a defense node and has independent decision-making and learning capabilities.

[0098] As an embodiment of the present invention, preferably, the global state space Indicates the entire network at time... The security situation is determined by multi-dimensional feature vectors. Composition. Among them, This represents a network traffic feature vector, including the traffic load of each defense node, inbound and outbound traffic rates, traffic protocol distribution, and number of session connections; This represents the attack alarm feature vector, including the number of system alarms, the distribution of alarm types, and the frequency of various attack alarms; This represents the system state feature vector, including the CPU utilization, memory utilization, and disk I / O utilization of each defense node; This represents a security posture feature vector, including the number of known vulnerable assets, the number of threat intelligence matching alerts, and the number of times users exhibited abnormal behavior.

[0099] As an embodiment provided by the present invention, preferably, the observation function Defined from global state To intelligent agents Local observation mapping relationship ,in To observe the noise, =1, 2, ..., N.

[0100] As an embodiment of the present invention, preferably, a local observation space Represents intelligent agents From the global state space Through observation function The local feature information of the agent obtained by mapping is represented as a subset of the defense node features in the global state space:

[0101] Specifically, this includes the local network traffic feature vector of the defense node. Local attack alert feature vector of defense node and defense node local system state feature vector .

[0102] As an embodiment of the present invention, preferably, the motion space Represents intelligent agents The set of executable defensive actions consists of multidimensional feature vectors. Composition. Among them... This indicates access control actions, including adjusting firewall rules and blocking a specific list of IP addresses; This indicates traffic management actions, including setting traffic limits and enabling traffic scrubbing; This indicates the detection response actions, including adjusting intrusion detection sensitivity and updating the attack signature database; This indicates deceptive deployment actions, including starting and stopping honeypot services and setting up bait data distribution; This indicates system configuration actions, including routing policy switching and service restarts.

[0103] As an embodiment of the present invention, preferably, the state transition function Defined in joint actions From the current global state Transition to the next global state probability distribution .

[0104] As an embodiment of the present invention, preferably, the reward function This indicates the intelligent agent. Instant rewards for performing a defensive action .

[0105] As an embodiment of the present invention, preferably based on Embodiment 1, the intelligent agent At any moment Execute action Instant rewards received afterward for:

[0106] ;

[0107] in, All are greater than or equal to 0. These are adjustable weighting coefficients;

[0108] For local rewards, the direct effect of the agent's own defensive actions is evaluated and represented as:

[0109] ;

[0110] in, To increase the success rate of interception, Indicates within the time window Inner agent Number of attacks successfully intercepted This indicates the total number of attacks detected within the time window; For resource consumption rate, Represents intelligent agents At any moment CPU and memory utilization Indicates baseline utilization. Indicates the maximum allowable utilization rate; For false alarm rate, This indicates the number of alarms that were confirmed as false alarms within the same time window. This indicates the total number of alarms within the same time window. All of these are weighting coefficients.

[0111] To incentivize cooperative behavior among agents, the cooperative reward is represented as follows:

[0112] ;

[0113] in, For action similarity, Represents intelligent agents The neighborhood group, Represents intelligent agents and his neighbors The action vector, This represents the similarity bandwidth parameter; For the number of strategy conflicts, This represents a conflict detection function that returns True when the actions of two agents directly contradict each other; where... For indicator functions; All of these are weighting coefficients.

[0114] The global reward, reflecting the overall security status of the network, is calculated by the central coordination layer based on the global state and distributed to each agent, and is represented as:

[0115] ;

[0116] in The change in global security score; global security score function. Traffic security is divided into... , Abnormal traffic ratio; alarm security score , The number of high-risk alerts. Alarm threshold; system security is divided into... , For defense nodes The number of resource items exceeding the limit, The total number of monitored resource items for each defense node; threat security is divided into... , The number of known vulnerable assets. Match the number of alerts to threat intelligence. Total assets Total number of threat intelligence entries; These are the weighting coefficients; For attack diffusion, This represents the number of newly attacked defense nodes added within the same time window, where N is the total number of defense nodes; These are the weighting coefficients.

[0117] Example 3:

[0118] As an embodiment of the present invention, preferably based on Embodiments 1 and 2, the local agent layer consists of various defense nodes distributed in the network environment. Each node acts as an agent, performing local reinforcement learning training based on its local observations. Each agent... Local policy training and experience collection are performed based on the Soft Actor-Critic (SAC) algorithm. The core network includes a policy network (Actor). Action Value Network (Critic) Target Value Network ,in, The parameters are the network parameters for the policy network and the value network, respectively. The network architecture for both is a multilayer perceptron, and the hidden layers use the ReLU activation function.

[0119] As an embodiment of the present invention, preferably, the agent constructs a local experience replay buffer by performing actions in the environment and observing the results. At each time step The agent performs the following process:

[0120] Obtain current local observations from the environment ;

[0121] According to the current strategy Select Action ;

[0122] Execute action Receive instant rewards and local observations at the next time step ;

[0123] empirical tuples Store to local experience replay buffer The buffer uses a first-in, first-out (FIFO) strategy to maintain a fixed capacity.

[0124] As an embodiment of the present invention, preferably, the agent maintains a local experience replay buffer and periodically samples experience data from the buffer, updating the action value network and policy network by minimizing the loss function; the method for local policy training based on the Soft Actor-Critic (SAC) algorithm includes:

[0125] First, perform network initialization, initializing the policy network parameters. Two action value network parameters And copy to obtain the corresponding target network parameters. ;

[0126] Perform empirical sampling from the buffer The random sampling batch size is empirical data ;

[0127] To calculate the target Q-value, for each sampling experience, the minimum strategy in the target network is used to calculate the target Q-value to suppress overestimation, expressed as:

[0128] ,in It is the next action sampled from the current strategy. Discount factor;

[0129] The action value networks are updated by minimizing the following loss function:

[0130] ;

[0131] The policy network is updated, and the goal of the policy network update is to minimize the loss function that combines Q-value estimation and policy entropy:

[0132]

[0133] in , Temperature parameters are used to control the intensity of entropy regularization, while temperature parameters It can automatically adjust based on strategy entropy to balance exploration and exploitation, as represented by: ,in Entropy of the current batch strategy The mean, Let the target entropy be a negative value of the action space dimension. The learning rate;

[0134] Perform soft updates to the target network to ensure training stability, as shown below:

[0135]

[0136] in, This is the soft update coefficient.

[0137] Example 4:

[0138] As an embodiment of the present invention, preferably based on embodiments one to three, the edge federation aggregation layer is a key intermediate layer connecting the local agent layer and the central coordination layer. Its core function is to achieve model collaboration and knowledge sharing among agents within a region while protecting data privacy and reducing communication overhead. This layer divides multiple agents into edge groups based on network topology and functional similarity. Each group has an edge aggregation node, which is responsible for periodically aggregating the local model parameters of the agents within the group to generate edge policy prototypes. First, N agents are divided into M edge groups. The grouping criteria include:

[0139] 1. Network topological proximity: Agents located in the same subnet or the same physical region are grouped together;

[0140] 2. Functional similarity: Node agents that perform similar defense functions, such as all being firewall nodes or all being intrusion detection nodes, are grouped together;

[0141] 3. Communication latency constraint: Ensure that the communication latency of nodes within the group is below a threshold. Define intelligent agents The edge group to which it belongs is Each edge group has a dedicated edge server deployed as the edge aggregation node for each group.

[0142] As an embodiment of the present invention, preferably, the edge aggregation node uses a fixed local training round. As a trigger condition, the method for the edge federated aggregation layer to perform periodic federated aggregation includes:

[0143] Multiple agents are divided into edge groups, and each edge group is assigned an edge aggregation defense node;

[0144] Parameters are collected, and the edge aggregation node distributes the data to all agents within the group. Each agent sends a model parameter request and responds by uploading its current policy network parameters. Furthermore, to protect privacy, policy network parameters must be homomorphically encrypted before being uploaded;

[0145] Assign aggregate weights to the model parameters of each agent Weights are calculated based on the amount of empirical data, recent defense effectiveness, and model quality evaluation, specifically using the amount of data in the agent's local experience buffer. This indicates the amount of experience data and the agent's performance within a preset time window. Average reward within This represents the recent defense effectiveness and the policy network loss function value of the agent model. This represents the model quality assessment, reflecting the degree of model convergence; the weight calculation formula is expressed as: ;

[0146] in, For temperature parameters;

[0147] ,in These are the maximum values ​​of the corresponding indicators within the group. These are the weighting coefficients for each dimension;

[0148] An edge strategy prototype is generated using a weighted linear fusion method. , represented as:

[0149] ;

[0150] The aggregated parameters are pruned to prevent parameter explosion, as shown below:

[0151] ;

[0152] in It is the parameter clipping threshold;

[0153] Each completed After rounds of local training, the edge aggregation node will generate an edge policy prototype. The message is sent to all local agents within the group. After receiving the message, the agents synchronize their local parameters through policy fusion.

[0154] ;

[0155] in, The strategy fusion weight coefficients.

[0156] As an embodiment of the present invention, preferably, the central coordination layer acts as the core scheduling and decision-making hub, not directly participating in local defense decisions. It performs global coordination by aggregating the policy information of each edge group, and is responsible for integrating the policy information of each edge group to generate a global meta-policy. The central coordination layer has a dedicated central server, serving as the central coordination node. The central coordination node maintains a global verification environment, which is a small-scale verification environment built based on a subset of de-identified historical data collected from each edge node, simulated attack traffic, and standardized benchmark test sets. The central coordination node aggregates data in rounds of edge federation. As a trigger condition, edge policy prototypes are collected from each edge aggregation node. The methods for generating periodic global meta-policies include:

[0157] The central coordination layer maintains a global verification environment, periodically collects edge policy prototypes from each edge aggregation defense node, and deploys the edge policy prototypes in the global verification environment;

[0158] In a global verification environment, edge policy prototypes are evaluated and screened based on a global security scoring function. The screened edge policy prototypes are then weighted and linearly fused to generate an initial global meta-policy. Specifically: edge policy prototypes are evaluated and screened; each prototype is deployed in the global verification environment; defense performance metrics are calculated based on security data generated during the testing period; and the global security scoring function from the global reward is used for calculation.

[0159] ;

[0160] in, For traffic security purposes, For alarm safety points, For system security purposes, The threat security score is calculated using the same formula as the global security scoring function, based on a preset performance threshold. Select edge models that meet performance standards and are stable. ,in This represents the performance variance across multiple tests. To establish a stability threshold, models with abnormal performance or excessive volatility were excluded; a weighted linear fusion method was used for the initial global meta-policy. Generate, represented as:

[0161] ;

[0162] Among them, weight , For temperature parameters;

[0163] The updated global meta-policy is obtained by fine-tuning the meta-learning based on the meta-learning framework. The method of fine-tuning the meta-learning based on the meta-learning framework is as follows: the defense environment of each edge group is regarded as a meta-task, and the global meta-policy is made capable of quickly adapting to different edge environments through inner loop task adaptation and outer loop global policy update. The meta-learning framework is the Model-Agnostic Meta-Learning framework.

[0164] As an embodiment of the present invention, preferably, meta-learning fine-tuning is performed based on the MAML (Model-Agnostic Meta-Learning) framework, enabling the global meta-policy to quickly adapt to different edge environments, and treating the defense environment of each edge group as a meta-task. The corresponding dataset is a representative subset of the local experience data of the marginal group. Define the task loss function as the average loss of the global meta-policy network on this dataset:

[0165] ;

[0166] First, perform inner loop adaptation by applying gradient descent to each meta-task to obtain task-specific adaptation parameters, expressed as:

[0167] ;

[0168] in Let be the learning rate for the inner loop. Then, based on the parameters adapted for each task, calculate the meta-learning task loss function and update the global meta-policy, expressed as:

[0169] ;

[0170] in As the outer loop learning rate, this update optimizes the global meta-policy to adapt quickly to various edge tasks. Simultaneously, to reduce computational complexity, a first-order MAML is used to simplify calculations, ignoring the second-order derivative term. .

[0171] As an embodiment of the present invention, preferably, the meta-game model treats the adversarial process between the multi-agent defense system and the attacker as a multi-role incomplete information game. First, a dynamically updated attack and defense payoff matrix is ​​constructed. The attacker is a virtual adversary, and its strategy space includes typical attack patterns such as port scanning, brute-force attacks, vulnerability exploitation, lateral movement, and data theft. This is obtained by clustering historical attack data and is represented as follows:

[0172] The defender is a joint defense system composed of multiple agents. Each agent can choose to perform basic defense actions such as monitoring, blocking, trapping, and disposal. The defense strategy space... Defined as a combination of defensive actions of each agent, expressed as:

[0173] Based on global rewards Constructing an attack and defense benefit matrix from the cumulative expected value Each element Indicates defense strategy Counterattack Strategies Net income is expressed as:

[0174] ;

[0175] The payoff matrix is ​​periodically updated based on real-time adversarial data, using the following formula:

[0176] ;

[0177] in For learning rate, This represents the increment of the latest observed payoff matrix;

[0178] To find the optimal policy distribution for the defender, the attack-defense game is modeled as a two-player zero-sum game, and its mixed-policy Nash equilibrium is solved. Let the probability distribution of the defender's mixed policy be... Probability distribution of attacker's mixed strategy Then the expected return function is:

[0179]

[0180] Based on this, a hybrid strategy Nash equilibrium is solved, where the defender's objective is to maximize their own gain, and the attacker's objective is to minimize the defender's gain, thus constituting a minimax problem:

[0181] ;

[0182] This problem can be transformed into a linear programming problem:

[0183]

[0184] The linear programming problem described above is solved efficiently using a mathematical solver, yielding the Nash equilibrium strategy. ;

[0185] Nash Equilibrium-based Strategy Dynamic role allocation is implemented, with core defense roles including monitoring, blocking, decoy, and response. Monitoring roles are responsible for traffic monitoring and anomaly detection; blocking roles are responsible for attack interception and access control; decoy roles are responsible for honeypot deployment and attack inducement; and response roles are responsible for traffic scrubbing and system recovery. A role adaptation vector is constructed for each agent. This indicates the degree of matching in the four core defensive roles, where each dimension corresponds to one defensive role, and the calculation method is as follows:

[0186] ;

[0187] in Hardware performance adaptability represents the degree of matching between the agent's hardware resources and its role requirements. For intelligent agents to roles Key hardware metrics scores For the role Hardware requirement threshold; Location fit represents the deployment location of an agent in the network topology, where For intelligent agents The network location coordinates, For the role The ideal position vector, For network hop distance, For temperature parameters; The historical performance score represents the historical effects of an agent performing similar roles, where... For intelligent agents Execution role The number of successes For the total number of allocations, This represents the average task completion time. The standard completion time threshold;

[0188] Nash equilibrium strategy Translated into a specific role allocation scheme, let the number of agents to be assigned the roles of monitoring, blocking, trapping, and handling be as follows: ,satisfy ,in Consider assigning only one role to each agent and setting a maximum number of roles for each agent. Constraints, using decision variables Represents intelligent agents Has an execution role been assigned? Optimize role allocation:

[0189]

[0190] The linear programming problem described above is solved efficiently using a mathematical solver, yielding the optimal role allocation scheme. .

[0191] Each completed After the round of global fusion, the central coordinating node will generate a global meta-policy. And role allocation scheme The policy is distributed to each edge node, which then further distributes it to agents within the group. Agents receive the global meta-policy. Then, local parameter synchronization is also performed through policy fusion:

[0192] ;

[0193] in For strategy fusion weight coefficients, The parameters are for the local agent that has already integrated the edge strategy prototype; at the same time, after receiving the role assignment scheme, the agent immediately switches to the designated role, updates the local execution logic and configuration parameters, and updates the cooperation relationship and communication protocol with neighboring agents to ensure that it meets the collaborative requirements of the new role. Finally, it reports the switch completion status to the edge node.

[0194] Example 5:

[0195] As an embodiment of the present invention, preferably based on embodiments one to four, in the online defense phase, the entire multi-agent defense system adopts a three-layer architecture of "local agent layer - edge federation aggregation layer - central coordination layer" to realize dynamic online defense game decision-making in a cyclical iterative manner. The online dynamic defense game decision-making process is as follows:

[0196] (1) Local decision-making and experience collection:

[0197] intelligent agent At the present moment Obtain local observation information According to the current policy network and current role Generate defensive actions Different roles use role preference vectors to weight the original actions output by the policy network to ensure that the actions conform to the role's function, represented as:

[0198] ;

[0199] in This is a role preference vector used to suppress or enhance specific types of actions. For example, monitoring a role primarily enhances the weight of monitored actions, while blocking a role enhances the weight of blocked actions. For exploration noise during the online phase;

[0200] intelligent agent Perform defensive actions When the network environment changes, an immediate reward consisting of local, collaborative, and global rewards is obtained. The intelligent agent will use the experience data from this interaction. Store in local experience replay buffer This serves as the foundation for subsequent strategy training.

[0201] (2) Continuous local learning and strategy updates:

[0202] intelligent agent Regularly from its local experience buffer A batch of historical experience data is randomly sampled, and the SAC algorithm is used for local training to update the policy network parameters. This process continuously optimizes its defense strategy by utilizing newly collected experience, thereby improving decision-making performance in local environments.

[0203] (3) Edge-layer federated aggregation and strategy fusion:

[0204] Each completed After local training, the edge federated aggregation layer performs the model aggregation process; each edge aggregation node collects the agent policy network parameters within the group. Based on the amount of experience data of each agent Recent defensive effectiveness and model quality assessment Calculate aggregate weights Weighted aggregation is performed to generate edge strategy prototypes for each edge group. ;

[0205] Edge strategy prototype The policy is distributed to all agents within the group, and the agents update their local parameters through policy fusion.

[0206] (4) Central-level overall coordination and role allocation:

[0207] Each completed After edge federation, the central coordination layer executes the global strategy generation, meta-game model construction, and dynamic role allocation processes; the central coordination node collects edge strategy prototypes from each edge aggregation node. Defense performance testing is conducted in a global verification environment. Edge models are selected based on the defense performance evaluation, and weighted aggregation is used to generate an initial global meta-policy. And based on the MAML framework, meta-learning fine-tuning is performed to obtain the updated global meta-policy. .

[0208] Based on the global meta-policy generation, the payoff matrix of the attack-defense game is constructed and updated. The offensive and defensive game is modeled as a two-player zero-sum game, and it is transformed into a linear problem to solve for its mixed-policy Nash equilibrium. To determine the optimal strategy distribution that the defense system should adopt when facing various attack strategies;

[0209] Dynamic role allocation is performed based on the solved Nash equilibrium strategy, including monitoring, blocking, trapping, and disposal roles. This is adapted to hardware performance. Location adaptability and historical performance rating Calculate the role fit of each agent for different defensive roles. The optimal role allocation scheme is obtained by constructing and solving the role allocation optimization problem. ;

[0210] The central coordinating node will generate a global meta-policy. And role allocation scheme The information is distributed to each edge aggregation node, which then further distributes it to agents within the group. The agents update their local parameters through policy fusion and switch to the designated defense role according to the new role allocation scheme. After the switch is completed, the agents report their status back to the edge aggregation nodes above them.

[0211] The entire process forms a complete closed loop of "agent decision-making - agent execution - local learning - edge aggregation - global collaboration - role switching". The entire multi-agent defense system achieves continuous strategy optimization and adaptive role allocation in a dynamic adversarial environment, enhancing global collaborative defense capabilities.

[0212] Example 6:

[0213] As the sixth embodiment of this invention, this invention provides a multi-agent dynamic defense game system based on federated reinforcement learning. The system is used to implement the methods described in embodiments one through five, and adopts a layered design, mainly consisting of four layers: a local agent layer (execution layer), an edge federated aggregation layer (coordination layer), a central coordination layer (decision layer), and a control and interaction layer (interaction layer). The local agent layer consists of agent units deployed at each network node, responsible for real-time defense decision-making based on local observations and local model training; the edge federated aggregation layer consists of edge servers deployed at the edge of the network region, responsible for privacy protection and periodic aggregation of agent models within the group; the central coordination layer consists of a central server deployed at the network center, responsible for global meta-policy generation, meta-game modeling, and dynamic role allocation; the control and interaction layer provides a human-computer interaction interface for system configuration and visual monitoring. The overall system framework is as follows: Figure 3 As shown.

[0214] The environment modeling module is used to model distributed network defense scenarios as a partially observable Markov environment based on multi-agent systems and to design a hierarchical hybrid reward function.

[0215] As an embodiment of the present invention, preferably, the local agent layer consists of heterogeneous defense nodes distributed in the network. The defense nodes, as agents, perform local policy training and experience collection through the Soft Actor-Critic (SAC) algorithm.

[0216] The edge federated aggregation layer, which consists of multiple edge aggregation defense nodes, is responsible for periodically federating the model parameters of the agents within the group to generate edge policy prototypes.

[0217] The central coordination layer, which consists of central coordination defense nodes, is responsible for integrating various edge strategy prototypes, generating global meta-policies and performing meta-learning fine-tuning, while constructing an offensive and defensive meta-game model and executing dynamic role allocation.

[0218] The online decision-making module is used to execute a closed-loop online defense process, enabling multiple defense nodes to learn adaptive strategies and conduct efficient collaborative defense in a dynamic adversarial environment.

[0219] As an embodiment of the present invention, preferably, the action space of the agent in the local agent layer includes: access control actions, traffic management actions, detection and response actions, deception deployment actions, and system configuration actions. When generating defense actions, the policy network output is weighted by the role preference vector to ensure that the actions conform to the current role function. Specifically:

[0220] Environmental Awareness and Observation Module: This module acquires raw security data in real time, including local network traffic, attack alerts, system resource usage, and security events, through data acquisition agents deployed at each defense node. It then extracts local observation vectors according to a preset observation function. The module supports multi-source data fusion and streaming processing to ensure the real-time nature and completeness of the observation information.

[0221] The policy network and decision-making module, based on a deep learning framework, implements the agent's policy network. It employs an Actor-Critic network constructed using a multilayer perceptron, comprising one policy network (Actor) and two action value networks (Critic). The module receives local observation vectors and outputs defensive actions based on the current policy network, supporting online inference and action execution. Simultaneously, the module supports a role preference vector weighting mechanism, adjusting the action output weights according to the agent's currently assigned defensive role.

[0222] The Local Experience Replay and Learning Module implements local experience storage and policy training for the agent. The module has a built-in FIFO experience replay buffer that stores experience tuples from the agent's interactions with the environment. The module periodically samples batch data from the buffer and updates the local policy network based on the SAC algorithm, including gradient calculation and parameter updates for the policy network and action-value network. Simultaneously, the module supports receiving policy prototypes from the edge layer and global meta-policies from the central layer for policy fusion and parameter synchronization.

[0223] As an embodiment of the present invention, preferably, the central coordination layer further includes:

[0224] The global verification environment module is used to test and evaluate the defense performance of edge strategy prototypes.

[0225] The game solving module is used to model the attack-defense meta-game model as a two-player zero-sum game and solve the mixed-strategy Nash equilibrium.

[0226] The role allocation module is used to solve for the optimal role allocation scheme based on role suitability and hybrid strategy Nash equilibrium.

[0227] As an embodiment of the present invention, preferably, the edge federation aggregation layer includes:

[0228] Edge Group Management Module: Based on network topology proximity, functional similarity, and communication latency constraints, the module divides agents into multiple edge groups and assigns an edge aggregation node to each group. The module supports node status monitoring and communication link management within each group.

[0229] Edge Policy Prototype Module: This module enables encrypted collection of model parameters for agents within an edge group and generates edge policy prototypes. Triggered by a fixed number of local training epochs, the module requests encrypted policy network parameters from agents within the group and calculates aggregation weights based on each agent's experience data volume, recent defense performance, and model quality evaluation. The module uses a weighted linear fusion method to generate edge policy prototypes and performs parameter pruning to prevent parameter explosion.

[0230] The policy distribution and synchronization module is responsible for securely distributing edge policy prototypes and global meta-policies and role allocation schemes from the central coordination layer to agents within the group, and receiving status feedback from the agents. The module supports breakpoint resumption and asynchronous update mechanisms to ensure the reliability and timeliness of policy synchronization.

[0231] As an embodiment of the present invention, preferably, the system includes:

[0232] Multiple heterogeneous defense nodes: These serve as local agent layers, used for local decision-making, experience collection, and local policy training. Each defense node is an agent.

[0233] Multiple edge aggregation defense nodes: These constitute the edge federation aggregation layer. Each edge aggregation defense node is connected to an edge group consisting of multiple agents to periodically aggregate the model parameters of the agents within the group to generate edge policy prototypes.

[0234] Central Coordination Defense Node: It constitutes the central coordination layer and communicates with all edge aggregation defense nodes. It is used to periodically aggregate the prototypes of each edge strategy to generate a global meta-strategy, and to construct a meta-game model and dynamically assign roles. The generated global meta-strategy and role assignment scheme are then distributed to each edge aggregation defense node.

[0235] As an embodiment of the present invention, preferably, the central coordination layer includes:

[0236] Global Meta-Policy Module: This module collects and filters edge policy prototypes and generates global meta-policies. Triggered by a fixed edge aggregation round, the module collects edge policy prototypes from each edge aggregation node and performs defense performance testing in a global verification environment. Based on the performance evaluation results, the module filters qualified models, performs weighted fusion to generate an initial global meta-policy, and then fine-tunes it using meta-learning based on the MAML framework to generate the final global meta-policy.

[0237] The Meta-Game Model and Role Allocation Module: This module implements the functions of constructing the attack and defense payoff matrix, solving the Nash equilibrium, and dynamically allocating roles. The module constructs the attack and defense payoff matrix based on the global reward and updates it dynamically. The module models the attack and defense game as a two-player zero-sum game and solves the mixed-strategy Nash equilibrium through linear programming. Simultaneously, based on the Nash equilibrium strategy and the agent's role suitability (including hardware performance suitability, position suitability, and historical performance rating), the module constructs and solves the role allocation optimization problem to generate the optimal role allocation scheme.

[0238] Policy Distribution and Synchronization Module: This module is responsible for distributing the generated global meta-policy and role allocation scheme to each edge aggregation node and receiving status feedback from the edge layer. The module supports breakpoint resumption and asynchronous update mechanisms to ensure the reliability and timeliness of policy synchronization.

[0239] As an embodiment of the present invention, preferably, the control and interaction layer includes:

[0240] System Management Module: This module provides a web-based management interface for system administrators to configure global parameters, including edge group division rules, trigger round settings, and role assignment thresholds. The module monitors and displays key indicators such as the operational status of each level, resource utilization, and defense performance in real time, and supports system health monitoring and anomaly alerts.

[0241] Human-Computer Interaction Module: This module provides a visual console that displays real-time defense posture, attack alerts, and agent role distribution. Operations personnel can use the interface to manually review alerts, mark false positives and missed positives, adjust policy priorities, and manually trigger role switching. The module supports multiple alert notification methods, including work orders and emails, enabling human-machine collaborative defense decision-making.

[0242] As an embodiment of the present invention, preferably, multiple heterogeneous defense nodes (such as firewalls, intrusion detection nodes, etc.) in a distributed network are modeled as multi-agents, and a three-layer federated learning architecture including a local agent layer, an edge federated aggregation layer, and a central coordination layer is constructed. In the local agent layer, each agent makes independent decisions based on partially observable environments, uses the Soft Actor-Critic algorithm for local training, executes defense actions based on local observations, and stores interaction experience in a local buffer for policy updates. In the edge federated aggregation layer, agents are divided into multiple edge groups according to network topology and function. Agents within each group periodically upload encrypted policy parameters to the edge aggregation node. Aggregation weights are calculated based on the amount of experience data, recent defense performance, and model quality evaluation of each agent. Weighted aggregation is then performed to generate edge policy prototypes for each edge group, which are then distributed to agents within the group to achieve policy collaboration within the group. In the central coordination layer, policy prototypes from each edge group are periodically collected, performance evaluation and edge model selection are performed in a global verification environment, weighted aggregation is used to generate an initial global meta-policy, and meta-learning fine-tuning is performed based on the MAML framework. Based on this, an attack-defense payoff matrix is ​​constructed, and the attack-defense game is modeled as a two-player zero-sum game, with the mixed-policy Nash equilibrium being solved. Furthermore, dynamic role allocation is performed according to the solved Nash equilibrium policy, including monitoring, blocking, trapping, and disposal roles, to obtain the optimal role allocation scheme. The central coordination node distributes the generated global meta-policy and role allocation scheme to each edge aggregation node, which further distributes them to agents within the group. Agents update their local parameters through policy fusion and switch to the designated defensive role according to the new role allocation scheme.

[0243] A multi-agent dynamic defense game theory method and system based on federated reinforcement learning is proposed. This method models the distributed network defense scenario as a partially observable Markov environment based on multiple agents and designs a hierarchical hybrid reward function. A three-layer architecture is constructed based on federated reinforcement learning: a local agent layer, an edge federated aggregation layer, and a central coordination layer. The local agent layer performs local training and experience collection based on the SAC algorithm. The edge federated aggregation layer periodically aggregates models of defense nodes within the group to generate edge policy prototypes. The central coordination layer selects and merges the edge policy prototypes to generate a global meta-policy and performs meta-learning fine-tuning. Based on this, a meta-game model is constructed and dynamic role allocation is performed. Finally, a distributed multi-agent defense system capable of adaptive policy learning and dynamic game collaboration is built, enabling autonomous adaptation and efficient collaborative proactive defense capabilities of multiple defense nodes in a partially observable, dynamically adversarial network environment.

[0244] Example 7:

[0245] Based on Examples 1 to 6, such as Figure 4As shown, the solution of this invention was deployed in a large enterprise. The large enterprise cloud network environment encompasses a complex network architecture including the headquarters data center, three remote branch offices, and a hybrid cloud environment of AWS and Azure. More than 50 heterogeneous defense nodes are deployed in this network, including web application firewalls, host-based intrusion detection systems, cloud security groups, and endpoint protection agents.

[0246] 1. System initialization and layered architecture setup:

[0247] Agent Modeling and Grouping: All 50 defense nodes in the network are modeled as intelligent agents with autonomous decision-making capabilities. Based on network topological proximity, functional similarity, and communication latency constraints, they are divided into 5 edge groups: Headquarters Data Center Group, Branch Office A Group, Branch Office B Group, AWS Cloud Resource Group, and Azure Cloud Resource Group. Each group is configured with one edge server as an aggregation node.

[0248] Environment and Policy Initialization: A lightweight environment awareness module is deployed for each agent to continuously collect information such as local traffic, alarms, and resource status, forming a standardized local observation vector. The policy network and value network parameters of all agents are randomly initialized. A global verification environment is deployed at the central coordination layer, loading anonymized historical attack traffic and standardized test case sets.

[0249] 2. Online dynamic defense game closed-loop operation:

[0250] Local real-time decision-making and experience collection: Based on current local observations (such as abnormal API call sequences captured by cloud WAF nodes) and their assigned roles (initially defaulting to "monitoring role"), each agent generates defensive actions through the policy network (e.g., "adding specific suspicious IPs to a temporary blocking list and checking the behavior of hosts within the same subnet"). After the action is executed, the environment provides immediate rewards (positive rewards for successful interception, negative rewards for conflicts with neighboring node actions), and the interaction experience is stored in the local experience buffer.

[0251] Continuous local training: After each agent accumulates a certain amount of data (such as 100 new experiences) in its local buffer, it immediately starts a round of local training based on the SAC algorithm to update the parameters of its policy network and value network, thereby achieving continuous optimization of the individual policy.

[0252] 3. Edge federation aggregation and intra-group knowledge sharing:

[0253] Periodic model aggregation: Edge federation is triggered every 10 rounds of local training (approximately 1000 experience points processed). The edge aggregation node requests the current policy network parameters from all agents in the group.

[0254] Weighted aggregation generates a strategy prototype: Edge nodes dynamically calculate differentiated aggregation weights based on each agent's recent defense performance (average reward), richness of empirical data, and model convergence. Agents with better performance and richer data have higher parameter weights. A weighted fusion method is used to generate an edge strategy prototype representing the latest collaborative defense knowledge of the group.

[0255] Policy distribution and fusion: The generated edge policy prototype is distributed to all agents within the group. Each agent then weights and fuses it with its own current parameters, quickly absorbing collective experience within the group to achieve policy synergy and performance improvement.

[0256] 4. Centralized overall coordination and dynamic role allocation:

[0257] Global meta-policy generation: After every 5 rounds of edge aggregation, the central coordination layer initiates global collaboration. The central server collects policy prototypes from each edge group, conducts batch performance testing and evaluation in a global verification environment, and selects prototypes that meet performance standards and are stable.

[0258] Meta-learning optimization: The selected prototypes are fused to generate an initial global meta-policy. This meta-policy is then fine-tuned using a model-independent meta-learning framework to enable it to quickly adapt to different edge group defense environments.

[0259] Constructing an offensive-defense game model and solving for the Nash equilibrium: The central layer constructs a payoff matrix for the attacker's strategy space (such as scanning, vulnerability exploitation, and data theft) and the defender's joint strategy space based on historical adversarial data. By solving the mixed-policy Nash equilibrium of this matrix, the optimal randomized strategy distribution of the defense system under the current threat situation is obtained.

[0260] Dynamic role allocation: Based on the solved Nash equilibrium strategy and combined with the real-time state of each agent (including hardware performance, network location, and historical role execution success rate), an optimization algorithm dynamically assigns each agent one of the four core defense roles: "monitoring," "blocking," "luring," and "disposal." For example, cloud server nodes with ample computing resources are assigned to the "luring role" to deploy high-interaction honeypots, while high-performance firewall nodes located at the network ingress are assigned to the "blocking role."

[0261] 5. Strategy and roles are executed synchronously:

[0262] Global policy distribution: The central coordination layer securely distributes the optimized global meta-policy and the new role allocation scheme to each agent through the edge aggregation layer.

[0263] Agent policy update and role switching: After receiving the global meta-policy, the agent performs policy fusion again to absorb global wisdom. At the same time, it immediately switches its working mode according to the new role instructions: updates action preferences (e.g., "blocking role" will increase the weight of interception actions), adjusts the cooperation protocol with surrounding nodes, and reports the ready status.

[0264] In this embodiment, the average threat detection and response time is significantly reduced from 4.2 hours to 28 minutes, improving efficiency by 89%. The first-time interception success rate of zero-day attacks in cloud environments is improved by 70%; the policy conflict rate between defense nodes is reduced from 15% to below 2%. Through dynamic role allocation and collaborative reward mechanisms, the interception success rate of cross-cloud and cross-regional collaborative attacks (such as lateral movement) is improved by 65%; intelligent scheduling based on Nash equilibrium improves the utilization rate of critical defense resources (such as advanced threat analysis computing power) by 40% during peak attack periods. At the same time, the federated learning architecture ensures that raw security data does not need to leave the domain, reducing overall network bandwidth consumption by 30% and fully meeting data privacy compliance requirements. The layered distributed architecture avoids the risk of single point of failure, and the overall system availability reaches 99.95%. The system can automatically adjust grouping, optimize policies, and reassign roles according to network topology changes and threat evolution, possessing strong self-evolution capabilities.

[0265] The hierarchical hybrid reward function design helps balance local optimization and global objectives, improving the adaptability and robustness of the strategy. This invention designs a hierarchical hybrid reward function for each agent, integrating local rewards, cooperative rewards, and global rewards. This allows the agent to optimize its own defense while also considering cooperative benefits with neighboring nodes and global security objectives, thereby avoiding local optimization from compromising overall performance and improving the adaptability and robustness of the strategy in dynamic adversarial environments.

[0266] This invention employs a three-layer collaborative defense architecture: a local agent layer, an edge federated aggregation layer, and a central coordination layer. This architecture facilitates efficient adaptive policy learning and dynamic game collaboration. The local agent layer performs distributed training and real-time decision-making, the edge layer performs encrypted aggregation of models within groups, and the central layer generates global meta-policies and dynamically assigns roles. This architecture reduces communication overhead, avoids single points of failure, and effectively protects the privacy of original data at each node, while simultaneously enabling efficient adaptive policy learning and dynamic game collaboration.

[0267] Introducing a meta-game model based on Nash equilibrium and a dynamic role allocation mechanism facilitates the shift from passive response to proactive, intelligent, and collaborative defense. This invention models the offensive and defensive confrontation as a two-player zero-sum game at the central coordination layer. By solving for a hybrid strategy Nash equilibrium, it guides the defense system in dynamically allocating specialized roles such as monitoring, blocking, trapping, and handling, thereby achieving intelligent scheduling of defense resources and strategies, and ultimately transforming from passive response to proactive, collaborative defense.

[0268] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0269] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-agent dynamic defense game method based on federated reinforcement learning, characterized in that, Includes the following steps: The distributed network defense scenario is modeled as a partially observable Markov environment based on multiple agents. A seven-tuple environment model is defined, comprising a set of agents, a global state space, a local observation space, an action space, a state transition function, an observation function, and a reward function. A hierarchical hybrid reward function is designed, which is obtained by weighted summation of local rewards, cooperative rewards, and global rewards. Local rewards are used to evaluate the direct effect of an agent's own defense actions and are calculated by weighting the interception success rate, resource consumption rate, and false alarm rate. Cooperative rewards are used to encourage cooperative behavior among agents and are calculated by weighting the action similarity and policy conflict number among agents. Global rewards are used to reflect the overall security status of the network and are calculated by weighting the change in global security score and attack diffusion degree. Immediate rewards are calculated using a hierarchical hybrid reward function. Construct a hierarchical collaborative defense architecture based on federated reinforcement learning. The defense architecture includes: The local agent layer consists of heterogeneous defense nodes distributed in the network. Each agent in the local agent layer is based on the Soft Actor-Critic algorithm and uses local observations to train local policies and collect experience. The agent maintains a local experience replay buffer and periodically samples experience data from the buffer. The action value network and policy network are updated by minimizing the loss function. The edge federated aggregation layer periodically aggregates the local model parameters of agents within the group to generate edge policy prototypes. The central coordination layer evaluates, filters, and merges various peripheral strategy prototypes to generate a global meta-policy and performs meta-learning fine-tuning. Based on the global meta-policy, an offensive-defensive meta-game model is constructed to solve the hybrid strategy Nash equilibrium. Dynamic role allocation is then performed based on the hybrid strategy Nash equilibrium. The methods for constructing the offensive-defensive meta-game model and executing the dynamic role allocation include: Construct and dynamically update the attack and defense benefit matrix; The attack-defense meta-game model is modeled as a two-player zero-sum game, and the mixed-policy Nash equilibrium is solved to obtain the optimal policy distribution of the defender. Based on hardware performance adaptability, location adaptability, and historical performance scores, the role adaptability of each agent to the four defensive roles of monitoring, blocking, trapping, and handling is calculated. Based on the hybrid strategy Nash equilibrium and the role suitability of each agent, the role allocation optimization problem is solved to obtain the optimal role allocation scheme; Based on the defense architecture, a closed-loop online dynamic defense process is executed. The closed-loop online dynamic defense process includes local decision-making and experience collection, continuous local learning and policy updates, edge-layer federated aggregation and policy fusion, and central-layer global coordination and role allocation in a cyclical iteration. The method for periodic federation aggregation in the edge federation aggregation layer includes: Multiple agents are divided into edge groups, and each edge group is assigned an edge aggregation defense node; The edge aggregation defense node requests and collects policy network parameters that have been homomorphically encrypted from the agents within the group; Based on the amount of empirical data, defense effectiveness, and model quality evaluation, the aggregate weight of each agent is calculated. An edge policy prototype is generated using a weighted linear fusion method, and the policy network parameters are pruned to prevent parameter explosion. The edge policy prototype is distributed to agents within the group, and the agents update their local parameters through policy fusion.

2. The multi-agent dynamic defense game method based on federated reinforcement learning according to claim 1, characterized in that; Methods for local policy training based on the Soft Actor-Critic algorithm include: Perform network initialization: initialize the policy network parameters and action value network parameters, and obtain the corresponding target network parameters; Randomly sample batches of experience data from the local experience replay buffer; For each sampling experience, the target Q value is calculated using the minimum strategy in the target network to suppress overestimation; Update the action value network and policy network by minimizing the loss function; Perform soft updates on the target network to ensure training stability.

3. The multi-agent dynamic defense game method based on federated reinforcement learning according to claim 1, characterized in that, The method for generating the global meta-policy includes: The central coordination layer maintains a global verification environment, periodically collects edge policy prototypes from each edge aggregation defense node, and deploys the edge policy prototypes in the global verification environment; In a global verification environment, the defense performance of edge policy prototypes is evaluated and screened based on a global security scoring function. The screened edge policy prototypes are then weighted and linearly fused to generate an initial global meta-policy. The updated global meta-policy is obtained by fine-tuning the meta-learning based on the meta-learning framework.

4. The multi-agent dynamic defense game method based on federated reinforcement learning according to claim 1, characterized in that, The closed-loop online dynamic defense process includes: Local decision-making and experience collection: The agent generates and executes defensive actions based on the current policy network and role preference vector, obtains immediate rewards consisting of local rewards, cooperative rewards and global rewards, and stores the experience in the local experience replay buffer; Continuous local learning and policy update: The agent periodically samples experience data from the buffer and performs local training based on the Soft Actor-Critic algorithm to update the policy; Edge layer federated aggregation and policy fusion: After each preset round of local training is completed, the edge aggregation defense node aggregates the policy parameters of the agents in the group to generate an edge policy prototype and distributes it. After receiving the edge policy prototype, the agents synchronize their local parameters. Central-level global coordination and role allocation: After completing each preset round of edge federation aggregation, the central coordination defense node aggregates the prototypes of each edge strategy to generate and fine-tune the global meta-strategy. At the same time, it constructs the attack and defense meta-game model and dynamically allocates roles. The updated global meta-strategy and role allocation scheme are then distributed to each agent, which updates its local parameters and switches roles.

5. A multi-agent dynamic defense game system based on federated reinforcement learning, characterized in that, The system is used to implement the method as described in any one of claims 1-4, the system comprising: The environment modeling module is used to model distributed network defense scenarios as a partially observable Markov environment based on multi-agent systems and to design a hierarchical hybrid reward function. In the local agent layer, the defense node acts as an agent and performs local policy training and experience collection through the Soft Actor-Critic algorithm. The edge federated aggregation layer, which consists of multiple edge aggregation defense nodes, is responsible for periodically federating the model parameters of the agents within the group to generate edge policy prototypes. The central coordination layer, which consists of central coordination defense nodes, is responsible for integrating various edge strategy prototypes, generating global meta-policies and performing meta-learning fine-tuning, while constructing an offensive and defensive meta-game model and executing dynamic role allocation. The online decision-making module is used to execute a closed-loop online defense process, enabling multiple defense nodes to learn adaptive strategies and cooperate in defense in a dynamic adversarial environment.

6. The multi-agent dynamic defense game system based on federated reinforcement learning according to claim 5, characterized in that, The action space of agents in the local agent layer includes: access control actions, traffic management actions, detection and response actions, deception deployment actions, and system configuration actions. When generating defense actions, the output of the policy network is weighted by role preference vectors.

7. The multi-agent dynamic defense game system based on federated reinforcement learning according to claim 5, characterized in that, The central coordination layer also includes: The global verification environment module is used to test and evaluate the defense performance of edge strategy prototypes. The game solving module is used to model the attack-defense meta-game model as a two-player zero-sum game and solve the mixed-strategy Nash equilibrium. The role allocation module is used to solve for the optimal role allocation scheme based on role suitability and hybrid strategy Nash equilibrium.

Images