Defensive strategy self-generation method and system for intelligent device cluster

Abstract

The application discloses a defense strategy self-generating method and system for a smart device cluster, and belongs to the technical field of network security of a smart device cluster. The system comprises a network modeling module, an attack strategy integration module and a defense strategy dynamic generation module. The method comprises the following steps: a network attack and defense game model is constructed to represent a network attack and defense scene of the current smart device cluster; the network topology of the smart device cluster is acquired, and an attack strategy and an action path are selected according to the network topology; attack behaviors are performed on the smart device cluster according to the attack strategy; the defense strategy dynamic generation module acquires a defense matrix of the current smart device cluster and the detection capability of each network node cluster, and a deep reinforcement learning model is used to generate a defense strategy. The application solves the problems of poor adaptability, insufficient cooperation and unreasonable resource scheduling in the traditional defense technology of the smart device cluster network, and greatly improves the overall efficiency of network security defense.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent device cluster network security, and particularly relates to a defense strategy self-generation method and system for an intelligent device cluster based on a partially observable Markov game (POMG) and deep reinforcement learning. BACKGROUND

[0002] With the rapid development of intelligent technology, intelligent device clusters are widely used in key fields such as industry, transportation, and energy. However, at the same time, network security problems are becoming increasingly serious, facing complex and diverse network threats such as APT, DoS attacks, and virus attacks. These attacks can lead to data leakage, system paralysis, and cause huge losses. Intelligent device clusters have complex structures and interconnected devices, and require multiple security defense resources to coordinate to resist threats. However, the existing defense mechanisms have obvious shortcomings: traditional fixed strategy defense methods are difficult to cope with changing attacks; each defense resource is independent and lacks coordination; and the existing defense resource configuration cannot flexibly adapt to the dynamic changes of intelligent device clusters. Therefore, it is extremely urgent to build a system that can adapt to the dynamic changes of intelligent device clusters and achieve autonomous optimization of defense strategies. Through innovative technology, intelligent generation of defense strategies and efficient allocation of resources are realized, which is of great significance to ensuring the network security of intelligent device clusters and improving their security and stability. SUMMARY

[0003] To solve the above technical problems, the present application makes in-depth research on the network security defense of intelligent device clusters and proposes a defense strategy self-generation method and system for intelligent device clusters based on partially observable Markov game and deep reinforcement learning technology, which solves the problems of poor adaptability, insufficient coordination, and unreasonable resource scheduling in traditional defense technology of intelligent device clusters, and greatly improves the overall efficiency of network security defense.

[0004] The defense strategy self-generation system for intelligent device clusters includes a network modeling module, an attack strategy integration module, and a defense strategy dynamic generation module.

[0005] The network modeling module, targeting the intelligent device cluster under current research, constructs a network attack-defense game model based on partially observable Markov games. This model includes: a participant space N, an action space A, a probability space P, a policy space S, a reward space R, and a state space M. The probability space P quantifies the probability distribution of the attacker's and defender's action choices under different policies; the policy space S contains the attacker's attack policy and the defender's dynamic defense policy; the reward space R contains the rewards obtained by the attacker and defender after each round of confrontation; and the state space M contains a defense matrix and an attack matrix. Each network node cluster in the intelligent device cluster is represented as a row in the matrix. Network node cluster types include data centers, communication node clusters, and peripherals. Each column of the matrix corresponds to a network node in the cluster. The element values ​​in the defense matrix correspond to the node's defense value, and the element values ​​in the attack matrix correspond to the node's attack value. The node's defense value is determined by quantifying its defense resources, representing the node's defensive capabilities. The node's attack value represents the intensity of the attack on the node, determined by quantifying attack resources.

[0006] The attack strategy integration module is used to set attack strategies and provide them to the network attack and defense game model, and select attack strategies and action paths according to the network topology of the intelligent device cluster.

[0007] When the intelligent device cluster engages in an attack and defense game, the system executes corresponding attack behaviors on the intelligent device cluster according to the attack strategy to obtain the current attack matrix of the intelligent device cluster; the defense strategy dynamic generation module obtains the current defense matrix of the intelligent device cluster and the quantified detection capabilities of each network node cluster, and generates defense strategies using a deep reinforcement learning model.

[0008] The system also includes a strategy execution and feedback module, which executes the defense strategy generated by the defense strategy dynamic generation module, updates the network state space and calculates the reward calibration matrix based on the attack and defense confrontation results, and feeds it back to the defense strategy dynamic generation module to drive the defense strategy update; the reward calibration matrix is ​​calculated based on the reward function of the action, which is calculated from the attack and defense results in the confrontation process and the resources consumed to execute the defense action.

[0009] The attack strategies set by the attack strategy integration module include: Denial-of-Service (DoS) attacks, Virus propagation, Advanced Persistent Threat (APT) attacks, vulnerability exploitation attacks, detection elimination attacks, and random control attacks. The detection elimination attack refers to performing actions that increase attack behavior or reduce the target's detection capability. The random control attack refers to performing actions that control the botnet host on the first execution and then performing a DoS attack on subsequent executions.

[0010] Accordingly, the present invention provides a method for self-generating defense strategies for intelligent device clusters, comprising the following steps:

[0011] Step 1: The network modeling module constructs a network attack-defense game model to represent the current network attack-defense scenario of the intelligent device cluster. The network attack-defense game model is represented as a six-tuple (N, A, P, S, R, M). The participant space N includes attackers and defenders, the action space A includes the attacker's attack actions and the defender's defense actions, the probability space P quantifies the probability distribution of the attacker's and defender's action choices under different strategies, the strategy space S includes the attacker's attack strategy and the defender's dynamic defense strategy, the reward space R includes the reward value obtained by the attacker and defender after each round of confrontation based on the attack-defense results, and the state space M includes the defense matrix and the attack matrix. Each network node cluster in the intelligent device cluster is taken as a row of the matrix. The network node cluster types include data centers, communication node clusters, and peripherals. Each column of the matrix corresponds to a network node in the network node cluster. The element values ​​in the defense matrix correspond to the node's defense value, and the element values ​​in the attack matrix correspond to the node's attack value. The node's defense value is determined by quantifying the node's defense resources, representing the node's defense capabilities. The node's attack value represents the intensity of the attack on the node, determined by quantifying the attack resources.

[0012] Step 2: Obtain the network topology of the smart device cluster, and select the attack strategy and action path based on the connection characteristics of the nodes in the network topology;

[0013] Step 3: The intelligent device cluster engages in an attack-defense game, including: executing corresponding attack behaviors on the intelligent device cluster according to the attack strategy, and outputting the current attack matrix of the intelligent device cluster; the defense strategy dynamic generation module obtains the current defense matrix of the intelligent device cluster and the quantified detection capabilities of each network node cluster, generates a defense strategy using a deep reinforcement learning model, drives the defense strategy update using a reward calibration matrix, and outputs the optimal defense action and defense matrix; the reward calibration matrix is ​​calculated based on the reward function of the action, which is calculated from the attack and defense results during the adversarial process and the resources consumed by executing the defense action.

[0014] In step 3, the deep reinforcement learning models set in the defense strategy dynamic generation module include DDQN, REINFORCE, and Actor-Critic. If the attack strategy is a multi-action phase attack, the DDQN algorithm is called to generate the defense strategy; if the attack strategy is a single-action regular attack, the Actor-Critic or REINFORCE algorithm is called to generate the defense strategy.

[0015] The advantages and positive effects of this invention are as follows:

[0016] (1) The method and system of this invention construct a network attack and defense game model covering multiple spaces, integrates multi-dimensional information such as network architecture and attack and defense resource distribution with typical attack strategies, and adopts a deep reinforcement learning algorithm. Through dynamic environment modeling and multi-stage attack identification, it realizes autonomous decision-making on defense strategies and optimizes the defense status. The method and system of this invention can provide dynamic security protection for intelligent device cluster systems and solve the problems of lagging defense strategies and excessive resource consumption in existing defense strategies.

[0017] (2) The method and system of this invention innovatively integrate a variety of typical attack strategies: By deeply analyzing the characteristics of virus attacks, DoS attacks, APT attacks and other attacks in intelligent device cluster networks, and combining them with the actual network topology, this invention constructs a comprehensive and realistic set of attack strategies, which makes the training and optimization of defense strategies more targeted and can effectively cope with diverse attack scenarios.

[0018] (3) The method and system of this invention utilize deep reinforcement learning to construct a dynamic generation mechanism for defense strategies: using POMG as a theoretical framework, it characterizes the strategic interaction between the attacker and defender in network attack and defense scenarios with incomplete information; with the help of deep reinforcement learning algorithms such as DDQN, REINFORCE, and Actor-Critic, it continuously optimizes the defense strategy based on real-time network state feedback and a carefully designed reward mechanism; the reward mechanism comprehensively considers factors such as the success or failure of defense and resource consumption, guiding the deep reinforcement learning algorithm to generate strategies that are more adaptable to complex attack environments. The method and system of this invention rely on deep reinforcement learning to construct a defense strategy generation mechanism with excellent dynamic adaptability. In network attack and defense scenarios, information is often incomplete, and traditional methods are difficult to effectively handle this uncertainty. This invention uses Partially Observable Markov Games (POMG) as a solid theoretical framework to accurately characterize the strategic interaction process between the attacker and defender in situations with incomplete information. It meticulously models key elements such as participants, actions, probabilities, strategies, rewards, and states in network attack and defense, clearly presenting the complex dynamic process of the attack and defense game. Simultaneously, advanced deep reinforcement learning algorithms such as DDQN (Double-DQN), REINFORCE, and Actor-Critic are introduced. These algorithms can continuously learn and optimize based on real-time network feedback and a carefully designed reward mechanism. The reward mechanism of this invention comprehensively considers multiple factors such as the success or failure of defense, the degree of resource consumption, and the severity of the attack, providing clear and reasonable optimization guidance for the algorithm. Through continuous iterative training, the deep reinforcement learning algorithm can generate defense strategies that are highly adaptable to complex and ever-changing attack environments, achieving a leap from static defense to dynamic and adaptive defense, significantly enhancing the ability of defense strategies to cope with different attack scenarios. Attached Figure Description

[0019] Figure 1 This is an application example diagram of the self-generated defense strategy system for intelligent device clusters of the present invention;

[0020] Figure 2 This is an example diagram illustrating the construction of state matrices for network topologies of varying complexity according to embodiments of the present invention;

[0021] Figure 3 This illustrates the correspondence between attack strategies and actions in embodiments of the present invention.

[0022] Figure 4 This is a diagram illustrating the interaction model of deep reinforcement learning and network attack-defense game environment in an embodiment of the present invention.

[0023] Figure 5 This is a diagram showing the input and output of deep reinforcement learning in an embodiment of the present invention.

[0024] Figure 6 This is a flowchart illustrating the attack and defense game between the method of this invention and a cluster of intelligent devices. Detailed Implementation

[0025] The exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can better understand the technical features and effects of the present invention.

[0026] This invention presents an autonomous defense strategy generation system for intelligent device clusters based on Partially Observable Markov Game (POMG) and deep reinforcement learning. The system mainly comprises an attack strategy integration module, a defense strategy dynamic generation module, and a network modeling module. The network modeling module, targeting the intelligent device cluster under study, constructs network attack and defense scenario models based on POMG, including abstract modeling of network elements, matrix mapping, and network topology analysis. The attack strategy integration module analyzes typical attack characteristics, designs and stores multiple attack strategies, providing strategy support for simulated attacks. The defense strategy dynamic generation module employs a deep reinforcement learning model to perform environment modeling, attack pattern recognition, and generate and optimize defense strategies.

[0027] like Figure 1As shown, the network modeling module abstracts the network attack and defense scenario using the six-tuple model NADGM. It constructs the six-tuple elements of the Network Attack and Defense Game Model (NADGM) through participant space construction units, action space construction units, probability space construction units, policy space construction units, reward space construction units, and state space construction units. Specifically, the participant space N includes attackers and defenders; the action space A defines the attacker's attack actions and the defender's defense actions; the probability space P quantifies the probability distribution of the attacker's and defender's action choices under different policies; the policy space S includes different attack policies of the attacker and dynamic defense policy combinations of the defender; the reward space R sets reward values ​​based on the adversarial attack and defense results of the attacker and defender, covering the rewards obtained by the attacker and defender after each round of adversarial attack and defense; and the state space M reflects the distribution of attack network resources and defense network resources of the attacker and defender through matrix mapping, with network resources such as server load. The state space M of this invention includes a defense matrix and an attack matrix. Each network node cluster in the intelligent device cluster is represented as a row in a matrix. Network node cluster types include data centers, communication node clusters, and peripherals. Each column of the matrix corresponds to a network node within a cluster. The elements in the defense matrix represent the node's defense value, and the elements in the attack matrix represent the node's attack value. A node's defense value is determined by quantifying its defense resources, characterizing its defensive capabilities. A node's attack value represents the intensity of an attack on the node, determined by quantifying attack resources.

[0028] The attack strategy integration module in this embodiment of the invention is responsible for storing and invoking six attack strategies, including DoS attacks (consuming bandwidth through traffic flooding) and APT attacks (long-term infiltration of critical nodes), and combines network topology design with attack node selection rules. For example, attackers can only infect the system through connection paths and cannot attack unconnected cluster nodes. The attack strategy integration module selects attack strategies and attack action paths based on the connection characteristics of nodes in the network topology of the intelligent device cluster.

[0029] The defense strategy dynamic generation module of this invention integrates deep reinforcement learning algorithms such as DDQN (Double Deep Q-Network), REINFORCE, and Actor-Critic, generating defense strategies through dynamic environment modeling and multi-stage attack identification. When a cluster of intelligent devices engages in attack and defense, corresponding attack behaviors are executed against the cluster according to the attack strategy, obtaining the current attack matrix of the cluster. The defense strategy dynamic generation module acquires the current defense matrix of the cluster and the quantified detection capabilities of each network node cluster, generating a defense strategy. The system also includes a strategy execution and feedback module, which executes the defense strategy generated by the dynamic generation module, updates the network state space based on the attack and defense results, and transmits feedback information to the dynamic generation module to achieve continuous optimization of the defense strategy. This invention utilizes a reward calibration matrix to drive defense strategy updates, outputting the optimal defense action and defense matrix. The reward calibration matrix is ​​calculated based on the reward function of the action, which is calculated from the attack and defense results during the adversarial process and the resources consumed in executing the defense action.

[0030] The present invention provides a method for self-generating defense strategies for intelligent device clusters based on POMG and deep reinforcement learning, which mainly includes four steps.

[0031] Step 1: Build a network attack and defense scenario model for intelligent device clusters based on POMG.

[0032] The network modeling module of this invention uses a network attack and defense game model to construct a six-tuple (N, A, P, S, R, M) for elements such as network attackers, defenders, actions, probabilities, strategies, rewards, and network states. It abstracts the intelligent device cluster network through a multi-dimensional matrix, establishes a coordinate mapping between the attack and defense matrix and the server, and designs matrices with different configurations based on the network topology to study the impact of network complexity on attack and defense.

[0033] like Figure 1As shown, in the action space A of this embodiment of the invention, the attacker's attack behaviors include four types: Enhance Attack, Attack Detection, Reduce Detection, and Control Zombie Hosts. This embodiment of the invention analyzes typical attack characteristics in intelligent device clusters and constructs six attack strategies, including Denial-of-Service (DoS) attacks, Virus propagation, Advanced Persistent Threat (APT) attacks, Vulnerability Exploitation attacks, Elimination Detection attacks, and Random Control attacks. It also sets attack node selection rules based on actual network attack scenarios and clarifies the specific actions of each attack strategy. Users can set attack strategies according to actual conditions.

[0034] In this embodiment of the invention, a DoS attack strategy is set. Represented as:

[0035] ;

[0036] Here, num_attack_actions represents the selectable range of attack strength, num_attack_positions represents the selectable range of reachable attack nodes, and Random represents random selection of operations.

[0037] Configure Elimination Detection attack strategy Represented as:

[0038] ;

[0039] Here, select means to choose. This indicates an increase in attack behavior. This indicates a reduction in target detection behavior.

[0040] Configure Random Control attack strategy Represented as:

[0041] ;

[0042] in, This indicates control over the behavior of the zombie host. This indicates a DoS attack.

[0043] Configure APT attack strategies Represented as:

[0044] ;

[0045] in, Represents reconnaissance and detection operations. This represents the minimum cost action during the attack phase.

[0046] Configure Virus attack strategy Represented as:

[0047] ;

[0048] in, This represents currently feasible virus attack operations. This represents the last selected virus attack.

[0049] Configure Vulnerability Exploitation attack strategy Represented as:

[0050] ;

[0051] in, Represents the vulnerability detection action, This represents a virus attack operation, which uses detected vulnerabilities to launch virus attacks.

[0052] Traditional network defenses often limit their simulation of attack strategies to common and known types, making it difficult to cope with the ever-evolving and complex attack methods in intelligent device cluster networks. This invention provides an in-depth and comprehensive analysis of the unique attack characteristics of various attacks, including virus attacks, DoS attacks, and APT attacks, in intelligent device cluster network environments. It not only meticulously studies the propagation mechanisms, infection paths, and destructive methods of virus attacks, but also conducts in-depth analysis of the traffic flooding patterns, attack source distribution, and impact on network bandwidth and server performance in DoS attacks. Furthermore, for APT attacks, it dissects the entire process from long-term infiltration to gradual penetration and precise strikes.

[0053] This invention provides an embodiment of the defender's defensive action space, comprising three actions: defense, patching, and enhanced defense. Defense actions are executed by isolating, blocking, and reallocating defensive resources. Patching increases the quantification of defensive resources. Enhanced defense increases detection capabilities, raising the probability of detecting the attacker in each round of actions. The defender's defense strategy is a dynamically combined sequence of defensive actions.

[0054] The state space M reflects the different state distributions of offensive and defensive resources in the intelligent device cluster system network, and the network constitutes a zero-sum game, satisfying... , hour, ;in Represents the defense matrix. Represents the attack matrix. () indicates the reward received by the defender. () indicates the reward received by the attacker. For the defender's state space, This represents the attacker's state space.

[0055] The attack and defense of an intelligent device cluster system constitute a zero-sum game. This invention models the intelligent equipment cluster network using a multi-dimensional matrix, constructs attack and defense matrices based on the network architecture, and establishes a matrix-server coordinate mapping. For example... Figure 2 As shown, embodiments of the present invention illustrate a single-layer network topology and a multi-layer network topology. Figure 2 The single-layer network topology in example (a) consists of data center 1, node cluster 2 and peripheral device 3. The established state matrix includes attack matrix and defense matrix. Each row in each matrix corresponds to a network node cluster in the network topology of the intelligent device cluster system. For example, the defense matrix contains 3 rows, which correspond to the network resources of data center 1, node cluster 2 and peripheral device 3 in turn. Figure 2 The multi-layer network topology in example (b) comprises data center 1, node clusters 2-7, and peripherals 8. The established state matrix includes an attack matrix and a defense matrix. Each row in each matrix corresponds to a network node cluster within the network topology. For example, the attack matrix has 8 rows, corresponding to the network resources of data center 1, node clusters 2-7, and peripherals 8, respectively. Each column of the matrix corresponds to different network nodes within the same network node cluster, and the element values ​​correspond to the node's attack or defense value. The rows and columns of the defense and attack matrices correspond to the same physical quantities: network node clusters and network nodes. The element values ​​in the defense and attack matrices are obtained based on physical quantities such as the defense and attack resources of network nodes in reality. In this embodiment, the node's defense value represents the node's defense strength, and the node's attack value represents the attack strength against the node. The specific defense and attack network resources of interest can be set according to actual conditions, such as server load, computing resources, or network bandwidth. Attack strength is a quantitative representation of the applied attack resources and attack effects, such as the degree of breach of node defenses. Defense strength is a quantitative representation of a node's defense resources; a higher value indicates a stronger defense capability. Defense resources and attack resources are quantified in the same way, and when defense and attack values ​​are the same, the required resource amounts are usually equal or nearly equal.

[0056] Step 2 involves acquiring the network topology of the smart device cluster. Attackers select attack nodes based on the single-layer or multi-layer characteristics of the network topology, prioritizing nodes with high reachability or high value. For example, in a single-layer topology, attackers might broadcast infection to all connected nodes, while in a multi-layer topology, they would gradually penetrate along hierarchical paths to the core nodes. Specifically, attackers assess node connectivity and the distribution of defense resources, such as the high bandwidth of server nodes and the low protection level of edge nodes, and combine this with pre-defined attack strategies, such as DoS attacks prioritizing bandwidth consumption or APT attacks focusing on long-term infiltration, to determine target nodes. This selection logic can be reflected through attack matrix mapping. Attacker actions also dynamically affect the defense resource status of nodes; for example, when attacking a node, its defense value decreases from 2 to 1, thus driving the evolution of the attack-defense game. The defense strategy module needs to be aware of this node selection behavior in real time, for example, by isolating attacked nodes or reallocating defense resources to high-risk areas to block attack paths and minimize losses.

[0057] Figure 3 This demonstrates the mapping between attack strategies and specific actions, and how these actions dynamically influence the state evolution of the network attack-defense game matrix. After selecting a target strategy from six options, the attacker deploys a series of consecutive actions to manipulate the game matrix. For example, the action of "reducing detection rate" adjusts the defender's detection capability matrix value from 2 to 1, thereby weakening its defense capabilities. Each attack strategy corresponds to a different combination of actions. For instance, a DoS attack strategy consumes bandwidth resources through traffic flooding, while an APT attack strategy gradually infiltrates key nodes through long-term infiltration. These actions reconfigure payoff parameters, such as the probability of attack success and the cost of defense, through strategic interactions, and establish an adaptive competitive environment. For example, a virus attack strategy may spread to multiple nodes through infection paths, while an Elimination Detection attack prioritizes attacking high-value nodes to disrupt the defense system. Figure 3 The study also demonstrates the correlation between attack strategies and network topology. For example, attackers need to choose their action path based on the connectivity characteristics of a single-layer or single-layer / multi-layer topology. This strategy-action-matrix linkage mechanism supports multi-dimensional scenario analysis, quantifies the impact of different attack behaviors on network status, and provides a data foundation for optimizing defense strategies. Figure 3 As shown, this example illustrates four attack actions and six attack strategies employed by the attacker. Figure 3 In the example, when attack strategy 1 (DoS) is executed to attack a node in a cluster with action 1 (enhanced attack), the corresponding row in the attack matrix represents the situation where the node in that cluster is attacked by this attack strategy-action combination, such as increasing the attack strength of a node to 1.

[0058] Step 3: Execute an attack and defense game among the intelligent device clusters and generate defense strategies through deep reinforcement learning.

[0059] Figure 4 This paper demonstrates a model that uses deep reinforcement learning (DRL) to generate defense strategies. Through dynamic environment modeling and multi-stage attack pattern recognition, it automatically generates adaptive defense strategies, enabling defenders to effectively respond to constantly evolving attack methods and possess real-time response capabilities. In the figure, the deep reinforcement learning model is used to dynamically generate strategies, providing defenders with adaptive strategies. Game theory-based network attack-defense game models form the foundation for deep reinforcement learning's participation in attack-defense game dynamics.

[0060] The dynamic generation module for defense strategies is based on the properties of zero-sum games and, in accordance with the principle of maximizing the minimum gain for the defender and the maximum gain for the attacker, constructs a multi-algorithm framework including DDQN, REINFORCE, and Actor-Critic algorithms, and sets the objective function for each algorithm. The reward function is designed based on the attack and defense resources in the network matrix and the resource consumption of defense actions, and the defense strategy is optimized through matrix transformation with reward calibration.

[0061] In this embodiment of the invention, the objective function of the DDQN algorithm is set. for:

[0062] ;

[0063] in, An immediate reward for the defender after performing an action; This is a discount factor used to balance the weights of current and future rewards, and its value is typically between 0 and 1. This represents the state of the intelligent device cluster, including a defense matrix and an attack matrix, indicating the current network environment. Defensive actions, i.e. defensive operations performed by the defender, are represented as a matrix with the same dimensions as the defense matrix; These are the parameters of the current Q-network. The parameters of the target Q-network are denoted by ; the Q-function Q() describes the state-action value relationship.

[0064] When using the REINFORCE algorithm, the defense policy is output through the policy network, and the policy network parameters are updated accordingly. The objective function for optimizing the defense strategy is as follows:

[0065] ;

[0066] in, For updating policy network parameters, These are the network parameters for the current policy. The learning rate controls the step size of each parameter update and determines the magnitude of policy adjustment. These are policy network parameters. The gradient operator is used to calculate the parameter update direction; Let be the policy network parameters at time t; It is the action-value function, representing the state at time t. Perform defensive actions Finally, the total expected future rewards are used to measure the payoff of the action-state combination; It is the policy function, representing the state given by the policy function. and parameters In this situation, take defensive action. The probability of.

[0067] When using the Actor-Critic algorithm to optimize defense strategies, the value network evaluates the value of actions, and the policy network outputs the probability of defense actions. The objective function of the Actor-Critic algorithm is defined as follows:

[0068] ;

[0069] ;

[0070] in, For the updated parameters of the value network Critic; The parameters for the current Critic; The learning rate for the Critic parameter; The time-series difference error at time t reflects the deviation between the value estimate and the actual return; It is the gradient operator for the value network parameter w; The action-value function evaluates the state. Perform defensive actions Expected return; The parameters for the updated policy network Actor; These are the parameters of the current policy network; The learning rate is the parameter of the policy network. The estimated action value at time t; Let be the policy function, representing the state. and parameters Take defensive action in the following situation The probability of.

[0071] The three deep reinforcement learning algorithms set in this embodiment of the invention are called as needed, based on the characteristics of the attack strategy, whether it is a multi-action attack or a single-action pattern attack. For example, DDQN is selected for complex multi-stage attacks, and Actor-Critic or REINFORCE is selected for single-action pattern attacks. The corresponding algorithm is called to generate the defense strategy as needed.

[0072] In incomplete information games, equilibrium solutions are achieved through minimax optimization, where both sides, under zero-sum game constraints, strategically maximize the attacker's minimum payoff and the defender's optimal strategy. The defender's payoff is shown below:

[0073] ;

[0074] Based on the properties of zero-sum games, the attacker's payoff is as follows:

[0075] ;

[0076] Therefore, we can derive the defender's strategy selection plan. As shown below:

[0077] ;

[0078] Attacker's strategy selection scheme It can be expressed by the following formula:

[0079] ;

[0080] By continuously optimizing their actions during the game, both attackers and defenders can achieve the optimal strategy under the current state and design. The strategy is determined when the following equation is satisfied: This constitutes a Nash equilibrium in a two-player zero-sum game.

[0081] .

[0082] The reward function is a key factor in evaluating the effectiveness of a policy in reinforcement learning algorithms. In adversarial processes, the reward function of an action... As shown in the following formula:

[0083] ;

[0084] in, This represents the defense value of the node in the nth row and mth column of the defense matrix. This represents the attack value of the node in the nth row and mth column of the attack matrix. This represents the resources required to perform a defensive action. An attack is considered successful if the defense value is less than the attack value; otherwise, the defense is considered successful. If a node successfully defends, it receives a +1 reward for the defensive action; if the attack succeeds, the defender is penalized with a -1 penalty. A reward function is calculated for each node in the smart device cluster. The value of is used to obtain the reward calibration matrix R of the intelligent device cluster under the current defense action.

[0085] This invention establishes a multi-algorithm deep reinforcement learning framework that generates an autonomous defense strategy by detecting and perceiving the matrix M and discrete actions A, and drives the defense strategy update with a reward calibration matrix R, outputting the optimal defense action and defense matrix. Figure 5 As shown, the input of the deep reinforcement learning model in this embodiment of the invention is a matrix combining the current defense matrix and detection capability of the intelligent device cluster. The detection capability is a quantified value of the attacker's behavior detected by the defender in each round; a stronger detection capability indicates a higher probability of detecting the attacker. In this embodiment, a detection capability value is quantified for each network node cluster in the intelligent device cluster, and these values ​​are merged into the last column of the defense matrix to obtain the input matrix. The detection capability of a network node cluster is generally related to the attack detection methods possessed by the cluster's devices. The detection capability value can be quantified based on the attack behaviors that the attack detection methods can detect. Therefore, in the attack-defense game of the intelligent device cluster, for the current attack matrix, the detection capability value can be used to determine whether the network node cluster has detected the attacker.

[0086] Deep reinforcement learning algorithms determine defensive actions based on input and the defensive action space. The game environment judges the effectiveness of the actions and outputs an adjusted defense matrix, which serves as the defense state matrix for the next round of attack and defense. For example... Figure 6 As shown, the attack-defense game process using the method of this invention in a smart device cluster is as follows: In the initial game, the attacker initiates an attack, and the defender selects a defensive action, all interacting within the attack-defense environment. During the game, it is first determined whether the attacker has been detected by the network node cluster. If not detected, the attacker executes an attack action, and then it is determined whether the data center has been compromised. If compromised, the attacker's reward is reduced by 1, and the defender's reward is increased by 1. The action reward is calculated based on the current defense matrix and attack matrix. The defender then selects a defensive action and updates the defense matrix. If not compromised, the defender selects a defensive action, executes it, updates the defense matrix, and then determines the attack-defense result based on the updated defense matrix and attack matrix, calculating the action reward. If the attacker is detected, the defender selects a defensive action, executes it, updates the defense matrix, and determines the attack-defense result based on the updated defense matrix and attack matrix, calculating the action reward. Attack and defense actions cycle through the attack-defense environment, with the reward determined based on the detection and breach results. This invention determines the attack and defense outcome based on defense and attack values. If the attack is successful, the attacker's gain is increased by 1 and the defender's gain is decreased by 1; if the defense is successful, the attacker's gain is increased by 1 and the defender's gain is decreased by 1. The defense strategy generation and optimization method of this invention incorporates the defender's action selection logic and utilizes algorithms such as reinforcement learning to allow the defender to dynamically select the optimal defense action in the attack and defense environment, participating in the game-like confrontation.

[0087] The above description is merely one embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention. Except for the technical features described in the specification, all other technologies are known to those skilled in the art. The present invention omits descriptions of well-known components and technologies to avoid redundancy and unnecessary limitation of the invention.

Claims

1. A self-generating defense strategy system for intelligent device clusters, characterized in that, It includes a network modeling module, an attack strategy integration module, and a defense strategy dynamic generation module; The network modeling module, targeting the intelligent device cluster under current research, constructs a network attack-defense game model based on partially observable Markov games. This model includes: a participant space N, an action space A, a probability space P, a policy space S, a reward space R, and a state space M. The probability space P quantifies the probability distribution of the attacker's and defender's action choices under different policies; the policy space S contains the attacker's attack policy and the defender's dynamic defense policy; the reward space R contains the rewards obtained by the attacker and defender after each round of confrontation; and the state space M contains a defense matrix and an attack matrix. Each network node cluster in the intelligent device cluster is represented as a row in the matrix. Network node cluster types include data centers, communication node clusters, and peripherals. Each column of the matrix corresponds to a network node in the cluster. The element values ​​in the defense matrix correspond to the node's defense value, and the element values ​​in the attack matrix correspond to the node's attack value. The node's defense value is determined by quantifying its defense resources, representing the node's defensive capabilities. The node's attack value represents the intensity of the attack on the node, determined by quantifying attack resources. The attack strategy integration module is used to set attack strategies and provide them to the network attack and defense game model, and select attack strategies and action paths according to the network topology of the intelligent device cluster. When the intelligent device cluster engages in an attack and defense game, the system executes corresponding attack behaviors on the intelligent device cluster according to the attack strategy to obtain the current attack matrix of the intelligent device cluster; the defense strategy dynamic generation module obtains the current defense matrix of the intelligent device cluster and the quantified detection capabilities of each network node cluster, and generates defense strategies using a deep reinforcement learning model.

2. The system according to claim 1, characterized in that, The attack strategies set by the attack strategy integration module include: Denial-of-Service (DoS) attacks, Virus propagation, Advanced Persistent Threat (APT) attacks, vulnerability exploitation attacks, detection elimination attacks, and random control attacks. The detection elimination attack refers to performing actions that increase attack behavior or reduce the target's detection capability. The random control attack refers to performing actions that control the botnet host on the first execution and then performing a DoS attack on subsequent executions.

3. The system according to claim 1, characterized in that, The system also includes a strategy execution and feedback module, which executes the defense strategy generated by the defense strategy dynamic generation module, updates the network state space and calculates the reward calibration matrix based on the attack and defense confrontation results, and feeds it back to the defense strategy dynamic generation module to drive the defense strategy update. The reward calibration matrix is ​​calculated based on the reward function of the action, which is calculated from the attack and defense results during the confrontation process and the resources consumed to perform the defensive action.

4. A method for self-generating defense strategies for intelligent device clusters, characterized in that, Includes the following steps: Step 1: The network modeling module constructs a network attack and defense game model to represent the current network attack and defense scenario of the intelligent device cluster; The network attack-defense game model is represented as a six-tuple (N, A, P, S, R, M). The participant space N includes attackers and defenders; the action space A includes the attacker's attack actions and the defender's defense actions; the probability space P quantifies the probability distribution of the attacker's and defender's action choices under different strategies; the strategy space S includes the attacker's attack strategy and the defender's dynamic defense strategy; the reward space R includes the reward values ​​obtained by the attacker and defender after each round of confrontation based on the attack-defense outcome; and the state space M includes the defense matrix and the attack matrix. Each network node cluster in the intelligent device cluster is represented as a row in the matrix. Network node cluster types include data centers, communication node clusters, and peripherals. Each column of the matrix corresponds to a network node in the network node cluster. The element values ​​in the defense matrix correspond to the node's defense value, and the element values ​​in the attack matrix correspond to the node's attack value. The node's defense value is determined by quantifying the node's defense resources, representing the node's defensive capabilities. The node's attack value represents the intensity of the attack on the node, determined by quantifying the attack resources. Step 2: Obtain the network topology of the smart device cluster, and select the attack strategy and action path based on the connection characteristics of the nodes in the network topology; Step 3: The intelligent device cluster engages in an attack-defense game, including: executing corresponding attack behaviors on the intelligent device cluster according to the attack strategy, and outputting the current attack matrix of the intelligent device cluster; the defense strategy dynamic generation module obtains the current defense matrix of the intelligent device cluster and the quantified detection capabilities of each network node cluster, generates a defense strategy using a deep reinforcement learning model, drives the defense strategy update using a reward calibration matrix, and outputs the optimal defense action and defense matrix; the reward calibration matrix is ​​calculated based on the reward function of the action, which is calculated from the attack and defense results during the adversarial process and the resources consumed by executing the defense action.

5. The method according to claim 4, characterized in that, In step 1, the constructed action space A includes four types of attacker behaviors: enhanced attack, attack detection, defense reduction, and control of zombie hosts. The constructed policy space S includes six attack policies: Denial-of-Service (DoS) attack, Virus propagation, Advanced Persistent Threat (APT) attack, Vulnerability Exploitation Attack, Detection Elimination Attack, and Random Control Attack. The Detection Elimination Attack refers to executing actions that increase attack behavior or decrease the target's detection capability. The Random Control Attack refers to executing the behavior of controlling the botnet host on the first execution and executing a DoS attack on subsequent executions.

6. The method according to claim 4, characterized in that, In step 1, the defensive behaviors of the defenders in the action space A are set to include three types: defense, patching, and enhancement. Among them, the patching behavior increases the quantitative strength of the defense resources, and the enhancement behavior increases the probability of detecting the attacker by increasing the node detection capability. The defense strategy is obtained by dynamically combining the defensive behaviors.

7. The method according to claim 4, characterized in that, In step 2, when selecting attack nodes for the action path, nodes with high reachability or high value are given priority as attack targets. The connectivity and defense strength of the nodes are evaluated, and the nodes to be attacked are determined in combination with the attack strategy.

8. The method according to claim 4, characterized in that, In step 3, the deep reinforcement learning models set in the defense strategy dynamic generation module include DDQN, REINFORCE, and Actor-Critic; if the attack strategy is a multi-action phase attack, the DDQN algorithm is called to generate the defense strategy. If the attack strategy is a single-action pattern attack, call the Actor-Critic or REINFORCE algorithm to generate a defense strategy.

9. The method according to claim 4, characterized in that, In step 3, the reward function for the defensive action is calculated. as follows: ； in, This represents the defense value of the m-th node in the cluster of the n-th network node in the defense matrix. This represents the attack value of the m-th node in the cluster of the n-th network node in the attack matrix; This represents the resources required to perform a defensive action; a successful attack is indicated by a defense value less than the attack value, otherwise a successful defense is indicated by a defense value. Calculate the reward function for each node in the smart device cluster. The value of is used to obtain the reward calibration matrix R for the current defensive action.

10. The method according to claim 8, characterized in that, In step 3, setting the objective function for each deep reinforcement learning model includes: (1) Objective function of DDQN algorithm ; in, As a reward for performing a defensive action, As a discount factor, For the defense matrix and attack matrix of intelligent device clusters, For the defensive behavior matrix, These are the parameters of the current Q-network. Here are the parameters of the target Q-network; Q is the Q-function. (2) The policy network using the REINFORCE algorithm outputs a defense strategy, and the defense strategy is optimized by updating the policy network parameters. The updated policy network parameters are... ; in, These are the network parameters for the current policy. The learning rate; These are policy network parameters. The gradient operator; Let be the policy network parameters at time t; It is the action-value function, representing the state at time t. Perform defensive actions Finally, the total expected future rewards; It is the policy function, representing the state given by the policy function. and parameters In this situation, take defensive action. The probability of; (3) The Actor-Critic algorithm is used to optimize the defense strategy. The objective function is set as follows: ； ； in, For the updated value network parameters, The parameters of the current value network, The learning rate for the value network parameters. Let be the time-series difference error at time t. It is the gradient operator of the value network parameter w. For action value functions; For the updated policy network parameters, These are the parameters of the current policy network. The learning rate is the parameter of the policy network. The estimated action value at time t; Let be the policy function, representing the state. and parameters Take defensive action in the following situation The probability of.

Images