Power system collaborative network attack modeling method and device

Abstract

The invention relates to the technical field of power system network space security, in particular to a power system collaborative network attack modeling method and device, and the method comprises the steps: carrying out the time-space collaborative hidden attack behavior of a plurality of attack modes under the partial observable condition of a power system, modeling a multi-agent part to observe a Markov decision process problem and generate a multi-tuple of a power system; a collaborative network attack model hidden under the observable condition of the system state part is constructed based on a multi-element system of the power system, and the multi-element system comprises at least one of an agent set, a system state space, a joint action space, a state transition probability, a reward function, a joint observation space, an observation probability function and a discount factor. Therefore, the problem that the attack effect cannot be guaranteed due to the fact that an attack model in the related technology cannot effectively describe various attack behaviors and the influence of information sharing on attack cooperation and effectiveness improvement during attack strategy optimization and execution is not considered is solved.

Description

Technical Field

[0001] This application relates to the field of power system cyberspace security technology, and in particular to a method and apparatus for modeling collaborative network attacks on power systems. Background Technology

[0002] Understanding the hidden and highly dangerous complex cooperative network attack behaviors in emerging power systems is fundamental to designing and deploying effective cooperative network attack detection measures. As cooperative network attacks in emerging power systems become increasingly complex in terms of spatiotemporal coordination across multiple attack types and locations, establishing cooperative network attack models that can effectively describe such complex attack behaviors is crucial.

[0003] In related technologies, research on collaborative network attack modeling mainly focuses on specific collaborative network attack problems, such as botnet collaborative attack models, which involve attackers controlling a large number of infected devices ("zombies") to form a distributed network and coordinating attacks through command and control servers.

[0004] However, the attack models in related technologies cannot effectively characterize various attack behaviors of multiple coordinated attack locations and / or attack methods. Furthermore, when constructing attack models, the observability and detection evasion of the system state are not considered, nor is the impact of attack strategy optimization and information sharing during execution on the improvement of attack cooperation and effectiveness. As a result, the attack effect cannot be guaranteed; improvements are urgently needed. Summary of the Invention

[0005] This application provides a method and apparatus for modeling collaborative network attacks in power systems, in order to solve the problems that attack models in related technologies cannot effectively characterize various attack behaviors, and do not consider the impact of attack strategy optimization and information sharing during execution on attack cooperation and effectiveness improvement, resulting in the inability to guarantee attack effectiveness.

[0006] The first aspect of this application provides a method for modeling cooperative network attacks on power systems, comprising the following steps: modeling the covert attack behavior of multiple attack methods in a spatiotemporal coordination under partially observable conditions of the power system as a multi-agent partially observable Markov decision process problem; generating a tuple of the power system based on the multi-agent partially observable Markov decision process problem; and constructing a covert cooperative network attack model under partially observable conditions of the system state based on the tuple of the power system, wherein the tuple includes at least one of an agent set, a system state space, a joint action space, a state transition probability, a reward function, a joint observation space, an observation probability function, and a discount factor.

[0007] Through the above technical solutions, the embodiments of this application can model the covert attack behavior of spatiotemporal coordination based on multiple attack methods as a multi-agent partially observable Markov decision process problem, and then construct a cooperative network attack model, so that the cooperative network attack model can comprehensively characterize various possible covert cooperative network attack behaviors under partially observable system state conditions; and design a gated communication-based reinforcement learning algorithm Gated-BiCNet to optimize the cooperative network attack strategy. By pruning redundant communication information, the effective data ratio of communication information between agents is increased, thereby improving the cooperation and attack effect between agents in the cooperative network attack process.

[0008] Optionally, in one embodiment of this application, it further includes: generating a communication-based reinforcement learning Gated-BiCNet algorithm for optimizing cooperative network attack strategies through centralized optimal policy learning and distributed execution strategies under interactive communication, so as to optimize the effective data ratio of communication information between agents using the communication-based reinforcement learning Gated-BiCNet algorithm.

[0009] Through the above technical solution, the embodiments of this application can utilize the generated communication-type reinforcement learning Gated-BiCNet algorithm to optimize the communication process between agents in a cooperative network attack strategy, thereby improving attack strength and efficiency.

[0010] Optionally, in one embodiment of this application, the Gated-BiCNet algorithm includes at least one Gated-Actor network and at least one Critic network. Each Gated-Actor network and each Critic network belongs to an agent. Each Gated-Actor network and each Critic network is implemented based on a bidirectional recurrent neural network. Each agent in the Gated-BiCNet inputs observation information into a gate, which filters redundant data from the input information. Each Gated-Actor network generates and executes an action based on the removed redundant information. Each Critic network evaluates the action's Q-value based on the observation information and then updates the network parameters.

[0011] Through the above technical solutions, the embodiments of this application can prune redundant communication information by setting a gating filtering mechanism, increase the effective data ratio of communication information between agents, thereby improving communication efficiency and enhancing the cooperation and attack effect between agents in the process of cooperative network attack.

[0012] Optionally, in one embodiment of this application, optimizing the effective data ratio of communication information between agents using the communication-type reinforcement learning Gated-BiCNet algorithm includes: generating the effectiveness probability corresponding to the observation information; training the effectiveness probability using at least one preset auxiliary task, so that if the communication message is valid, it is input into the Actor network; otherwise, it is determined that there is redundant data in the communication message, so that the information obtained by multiplying the vector of the communication message with the zero vector is input into the Actor network.

[0013] Through the above technical solution, the embodiments of this application can compare the Q-value of the action generated based on the communication message and the Q-value of the action generated based on the local message by judging the value of the validity probability, and then determine whether message pruning is needed for the communication message, so as to ensure that valid communication information is input into the Actor network, thereby filtering redundant information and increasing the proportion of valid communication information between agents.

[0014] Optionally, in one embodiment of this application, the training formula for the validity probability is:

[0015]

[0016] Among them, L(o i Y(o) is the loss function to be minimized. i ) is the label for auxiliary tasks, p(o i ) represents the gating probability, o i For the observation of agent i, The expectation symbol represents the expectation of o. i The expected value is calculated from the probability distribution.

[0017] Through the above technical solution, the embodiments of this application can optimize the training of the validity probability model by minimizing the loss function, effectively improving its prediction accuracy, thereby ensuring the input of valid communication messages and filtering redundant invalid information.

[0018] A second aspect of this application provides a power system cooperative network attack modeling apparatus, comprising: a first modeling module, used to model the covert attack behavior of multiple attack methods in a spatiotemporal coordination under partially observable conditions of a power system as a multi-agent partially observable Markov decision process problem; a generation module, used to generate a tuple of the power system based on the multi-agent partially observable Markov decision process problem; and a second modeling module, used to construct a covert cooperative network attack model under partially observable conditions of the system state based on the tuple of the power system, wherein the tuple includes at least one of an agent set, a system state space, a joint action space, a state transition probability, a reward function, a joint observation space, an observation probability function, and a discount factor.

[0019] Through the above technical solutions, the embodiments of this application can model the covert attack behavior of spatiotemporal coordination based on multiple attack methods as a multi-agent partially observable Markov decision process problem, and then construct a cooperative network attack model, so that the cooperative network attack model can comprehensively characterize various possible covert cooperative network attack behaviors under partially observable system state conditions; and design a gated communication-based reinforcement learning algorithm Gated-BiCNet to optimize the cooperative network attack strategy. By pruning redundant communication information, the effective data ratio of communication information between agents is increased, thereby improving the cooperation and attack effect between agents in the cooperative network attack process.

[0020] Optionally, in one embodiment of this application, it further includes: an optimization module, used to generate a communication-based reinforcement learning Gated-BiCNet algorithm for optimizing cooperative network attack strategies through centralized optimal strategy learning and distributed execution strategies under interactive communication, so as to optimize the effective data ratio of communication information between agents using the communication-based reinforcement learning Gated-BiCNet algorithm.

[0021] Through the above technical solution, the embodiments of this application can utilize the generated communication-type reinforcement learning Gated-BiCNet algorithm to optimize the communication process between agents in a cooperative network attack strategy, thereby improving attack strength and efficiency.

[0022] Optionally, in one embodiment of this application, the Gated-BiCNet algorithm comprises at least one Gated-Actor network and at least one Critic network, each Gated-Actor network and each Critic network belonging to an agent, each Gated-Actor network and each Critic network being implemented based on a bidirectional recurrent neural network, and each agent in the Gated-BiCNet inputs observation information into a gate, the gate filters redundant data from the input information, each Gated-Actor network generates and executes an action based on the removed redundant information, and each Critic network evaluates the action's Q-value based on the observation information and then updates the network parameters.

[0023] Through the above technical solutions, the embodiments of this application can prune redundant communication information by setting a gating filtering mechanism, increase the effective data ratio of communication information between agents, thereby improving communication efficiency and enhancing the cooperation and attack effect between agents in the process of cooperative network attack.

[0024] Optionally, in one embodiment of this application, the optimization module includes: a generation unit for generating validity probabilities corresponding to observation information; and an input unit for training the validity probabilities using at least one preset auxiliary task, so that if the communication message is valid, it is input into the Actor network; otherwise, it is determined that there is redundant data in the communication message, so that the information obtained by multiplying the vector of the communication message with the zero vector is input into the Actor network.

[0025] Through the above technical solution, the embodiments of this application can compare the Q-value of the action generated based on the communication message and the Q-value of the action generated based on the local message by judging the value of the validity probability, and then determine whether message pruning is needed for the communication message, so as to ensure that valid communication information is input into the Actor network, thereby filtering redundant information and increasing the proportion of valid communication information between agents.

[0026] Optionally, in one embodiment of this application, the training formula for the validity probability is:

[0027]

[0028] Among them, L(o i Y(o) is the loss function to be minimized. i ) is the label for auxiliary tasks, p(o i ) represents the gating probability, o i For the observation of agent i, The expectation symbol represents the expectation of o. i The expected value is calculated from the probability distribution.

[0029] Through the above technical solution, the embodiments of this application can optimize the training of the validity probability model by minimizing the loss function, effectively improving its prediction accuracy, thereby ensuring the input of valid communication messages and filtering redundant invalid information.

[0030] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the power system cooperative network attack modeling method as described in the above embodiments.

[0031] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described power system cooperative network attack modeling method.

[0032] A fifth aspect of this application provides a computer program product that stores a computer program that, when executed by a processor, implements the above-described power system cooperative network attack modeling method.

[0033] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0034] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0035] Figure 1 This is a flowchart of a power system cooperative network attack modeling method provided according to an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of the structure of the communication-based reinforcement learning Gated-BiCNet algorithm for optimizing cooperative network attack strategies according to a specific embodiment of this application;

[0037] Figure 3 This is a schematic diagram of a gating mechanism framework for the Gated-BiCNet algorithm according to a specific embodiment of this application;

[0038] Figure 4 This is a schematic diagram of a power system cooperative network attack modeling device according to an embodiment of this application;

[0039] Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0040] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0041] The following describes a power system cooperative network attack modeling method and apparatus according to embodiments of this application with reference to the accompanying drawings. Addressing the problems mentioned in the background section regarding the inability of attack models to effectively characterize various attack behaviors and the failure to consider the impact of attack strategy optimization and information sharing during execution on attack cooperation and effectiveness, thus compromising attack effectiveness, this application provides a power system cooperative network attack modeling method. In this method, based on the spatiotemporal cooperative covert attack behaviors of various attack methods, a multi-agent partially observable Markov decision process problem can be modeled, thereby constructing a cooperative network attack model. This model comprehensively characterizes various possible covert cooperative network attack behaviors under partially observable system state conditions. Furthermore, a gated communication-based reinforcement learning algorithm, Gated-BiCNet, is designed to optimize the cooperative network attack strategy. By pruning redundant communication information, the effective data ratio of communication information between agents is increased, enhancing the cooperation and attack effectiveness between agents during the cooperative network attack process. Therefore, this method solves the problems of attack models in related technologies failing to effectively characterize various attack behaviors and failing to consider the impact of attack strategy optimization and information sharing during execution on attack cooperation and effectiveness, thus compromising attack effectiveness.

[0042] Specifically, Figure 1 This is a flowchart illustrating a power system cooperative network attack modeling method provided in an embodiment of this application.

[0043] like Figure 1 As shown, the power system cooperative network attack modeling method includes the following steps:

[0044] In step S101, the covert attack behavior of multiple attack methods in spatiotemporal coordination under partially observable conditions of the power system is modeled as a multi-agent partially observable Markov decision process problem.

[0045] Understandably, partial observable characteristics of a power system refer to the phenomenon where the system's operating status cannot be fully, in real-time, and accurately monitored and estimated due to the limited number of measuring devices, communication constraints, or data quality issues. This characteristic poses a significant challenge to the security monitoring of power systems and also provides opportunities for attackers.

[0046] Therefore, under partially observable conditions of the power system, attackers can bypass traditional monitoring methods through spatiotemporally coordinated covert attacks, posing a serious threat to the operation of the power grid.

[0047] Spatiotemporal coordinated covert attacks refer to attacks launched by attackers through carefully designed time-series and spatial distribution strategies at different locations and time periods within a power system. These attacks fully utilize some observable characteristics and cyber-physical coupling characteristics of the power system. Such attacks typically combine multiple methods such as False Data Injection (FDI), delay attacks, and topology attacks, and are optimized in both the time dimension (staged attacks) and the spatial dimension (multi-node coordination) to maximize destructiveness and minimize the probability of detection.

[0048] Therefore, in actual implementation, the embodiments of this application can fully consider the various possible covert collaborative network attack behaviors under partially observable conditions of the power system, such as spoofed measurement data injection attacks and low-rate denial of service (LDoS) attacks, and model them as a multi-agent partially observable Markov decision process (MA-POMDP) ​​problem. Then, a new power system collaborative network attack model can be constructed based on the multi-agent partially observable Markov decision process.

[0049] The technical solutions described in this application embodiment enable the final constructed system network attack model to comprehensively characterize various possible covert and coordinated network attack behaviors, thereby improving the comprehensiveness of the attack.

[0050] In step S102, a tuple of the power system is generated based on the multi-agent partially observable Markov decision process problem.

[0051] In this context, the tuple representation of a power system can be understood as abstracting the key elements of the power system and their interrelationships into a mathematical structure through formal modeling methods, in order to support accurate analysis and algorithm design.

[0052] As one possible approach, embodiments of this application can model the multi-agent partially observable Markov decision process problem of cooperative network attack as an 8-tuple:

[0053]

[0054] In the formula, A collection of intelligent agents; For the system state space; For joint action space; The state transition probability; For the reward function; Ω represents the joint observation space; γ represents the observation probability function; and γ∈[0,1] represents the discount factor.

[0055] Among them, regarding the set of intelligent agents

[0056] For network attacks involving multiple coordinated attack methods and time-coordinated network attacks, each agent is assigned to execute one type of attack. For spatially coordinated network attacks, each agent is assigned to launch the same type of attack from a specific location. For spatiotemporally coordinated network attacks, each agent is assigned to launch different types of attacks from a specific location. During the attack, the agents communicate with each other and maintain close cooperation to ensure the effectiveness of the attack. The definition is as follows:

[0057]

[0058] In the formula, i a N represents the number of agents that attack a specific location or carry out a specific attack.

[0059] Regarding the system state space S:

[0060] The system state space describes the global state of the power system. Selecting appropriate power system state information based on the attack target and its impact is crucial to the effectiveness of the attack. At time t, the system state space... The definition is as follows:

[0061] s(t)=[s 1 (t),s 2 (t),...,s k (t)] (3)

[0062] In the formula, s k (t) represents complete information about the system state, such as the active power, voltage, and current of all loads on all buses in the system. Appropriate system state information should be selected for different attacks.

[0063] Regarding joint observation space

[0064] Consider a more realistic partially observable scenario, where the attacker can only observe local system state information within their control. Similar to the selection of system state information, the appropriate observation information should be chosen based on the characteristics of the attack and influencing factors. At time t, agent i's observation o... i (t) is defined as:

[0065]

[0066] In the formula, This refers to partial information about the system state that agent i can observe, such as the active power, voltage, and current of loads on some buses in the system. Appropriate observation information should be selected for different attacks.

[0067] At time t, the joint observation of all agents Defined as:

[0068]

[0069] Regarding the space for joint action

[0070] During the attack, each agent i injects an attack vector into its target. Therefore, at time t, the action a of agent i is... i (t) is defined as:

[0071]

[0072] At time t, the joint action of all agents Defined as:

[0073]

[0074] To ensure the effectiveness and stealth of coordinated network attacks, a subset of the joint action set can be defined based on the attack effect and detection evasion, providing a basis for setting rewards under different subsequent actions. All joint actions satisfying constraints (8) and (9) are defined as subset a. T The set of joint actions without offensive behavior is defined as subset a. I .

[0075] f d (a(t))<ω d (8)

[0076] In the formula, f d (a(t)) represents the abnormal deviation of the system caused by action a(t); ω d The threshold for attack detection devices to detect attacks.

[0077] f k (a(t))≥ω k (9)

[0078] In the formula, f k (a(t)) represents the attack effect caused by action a(t); ω k The state of the target object to achieve the desired effect of the attack.

[0079] Regarding the reward function

[0080] Let all agents share a global reward R global At the same time, each agent has a local reward. The reward for agent i can be obtained by combining the two, as shown in formula (10).

[0081]

[0082] R global =αE+βC (11)

[0083] In the formula, E represents the attack effect, as shown in formula (12); C represents the degree of cooperation of the agents, as shown in formula (13); α and β are weights.

[0084] E = f k (a(t))-ω k (12)

[0085]

[0086] In the formula, S represents the degree of information sharing; the higher the degree to which agents share effective information, the better their cooperation. c To ensure task complementarity, different agents undertake tasks of varying importance, and the agents cooperate to achieve the overall goal; w1 and w2 are the weights.

[0087]

[0088] In the formula, To maximize the reward for the local attack effect of agent i; P i α' is the penalty for agent i being detected as abnormal by the attack detection device; β' and α' are the weights.

[0089] In the above embodiments, the multi-agent partially observable Markov decision process problem is modeled as an 8-tuple. This tuple modeling method can clarify the relationship between the elements and support multi-dimensional coupling analysis of various information, providing good data support for subsequent collaborative network attack models.

[0090] In step S103, a cooperative network attack model that is hidden under the condition that the system state is partially observable is constructed based on the tuple of the power system. The tuple includes at least one of the following: agent set, system state space, joint action space, state transition probability, reward function, joint observation space, observation probability function, and discount factor.

[0091] In one embodiment of this application, the aforementioned cooperative network attack model can be a cooperative attack model for injecting fake measurement data and low-rate denial-of-service attacks targeting novel power systems, referred to as the FDI-LDoS cooperative attack model, which includes the following parts:

[0092] FDI-LDoS coordinated attacks can launch False Data Injection (FDI) and Low-Rate Denial of Service (LDoS) attacks on measurement data from different locations in the power grid, aiming to maximize the overload of transmission lines or generators. The attacker launches an FDI attack on measurement data from some locations under their control, and an LDoS attack on measurement data from the remaining locations. This prevents the monitoring center from obtaining timely and accurate system operation data, thereby reducing the accuracy of Bad Data Detection (BDD). The monitoring center then makes incorrect scheduling decisions, ultimately causing transmission line or generator overload. The FDI-LDoS coordinated attack behavior is represented as: z a,d =z d +a, where z d The remaining measurement data that reaches the monitoring center after the LDoS attack, a = [a1, a2, K, a m ] T The injection vector for the FDI attack. The mathematical model of the FDI-LDoS cooperative attack is shown in formulas (15)-(28), where formula (15) is the attack target and formulas (16)-(28) are the constraints.

[0093] max D a =w g D g +w b D b (15)

[0094]

[0095] N a <N max (twenty one)

[0096]

[0097] In the formula, D a For FDI-DoS coordinated attack effects; D g and D b These represent the number of generator overloads and transmission line overloads caused by the FDI-LDoS coordinated attack, respectively; w g and w b D respectively g and D b The weights; and Δp represents the power generation of generator g and the flow rate of line l during the current time period, respectively, determined by the state estimation results; g and Δfl These represent the increase in power generation by generator g and the increase in flow rate by line l, respectively, for the next time period; I G L and L represent the number of generators and the number of transmission lines, respectively; and These are the upper and lower limits of the generator g's output power, respectively; These are false measurements; is the measurement matrix after suffering an FDI-LDoS attack; a is the FDI attack vector; N max The maximum number of nodes that can be attacked, z i It is a vector of measurement values. These are the measurement data that were delayed or discarded, γ z yes The largest proportion, T i t This represents the congestion level of node i at time t. This represents the attack cost of node i at time t. Let C be the attack cost coefficient for node i. max This represents the maximum attack cost that an attacker can incur. It is node N a The spectrum of abnormal and normal traffic mixed together. For node N a Acceptable energy distribution threshold; ω0 is the fundamental frequency; f0 is the zero frequency; a0 is the average value of the signal f(t); a n b is the amplitude of the cosine component; n The amplitudes of the sinusoidal component are E, τ, and T. X These represent the pulse amplitude, pulse width, and period of the signal f(t), respectively; σ f E refers to the average spectral energy at any frequency point f within the main lobe; f It is the energy at frequency f; E is the total energy at all frequency points in the main lobe. Let ω be the vector of estimated values ​​for the i1th group; r and ω p These represent the trust weights of the monitoring center for actual data and predicted data, respectively; K and P are positive integers. This is the acceptable residual threshold for the system.

[0098] The FDI attack model is as follows:

[0099] FDI attacks exploit vulnerabilities in malicious data detection methods, allowing attackers to inject incorrect data into a set of measurements, modifying the measurement vector z to z0. a = z + a. Where, the FDI attack vector a = [a1, a2, K, a m ] TThis is a spoofed data vector added to the original measurement z. If an attacker has complete knowledge of the system model (i.e., matrix H) and is capable of tampering with specific measurements, the FDI attack vector a = Hc, where c = [c1, c2, K, c...]. n ] T And even if it's not zero, it can still remain hidden. The tampered measurement value z a Change to z a =H(x+c)+e, which causes the state estimation to be affected by the degree of c, while the residual of the bad data detection check remains unchanged. At this time, the bad data detection method based on residuals cannot detect false data. Attackers can modify the measured values ​​and state variables to arbitrary values, which will endanger the safe and stable operation of the power system.

[0100] The goal of an FDI attack is to cause the monitoring center to misjudge the system state, execute incorrect scheduling decisions, and remain hidden from detection of malicious data, thereby overloading generators or transmission lines. The mathematical model for an FDI attack is as follows:

[0101] max D a fdi =w g fdi D g fdi +w b fdi D b fdi (29)

[0102]

[0103] In the formula, D a fdi Represents the effect of an FDI attack; D g fdi and D b fdi These represent the number of generator overloads and transmission line overloads caused by FDI attacks, respectively.

[0104] The specific model of an LDoS attack is as follows:

[0105] Distributed LDoS attacks can take the form of aperiodic aggregation to reduce the likelihood of detection and defense. Attackers instruct each attacker in the botnet to send aperiodic pulses by sending different instructions to bot agents, which eventually converge at the victim to form a periodic attack, then immediately cause damage. Therefore, aperiodic aggregated DLDoS attacks can enhance the stealth of the attack while maintaining its harmfulness.

[0106] The four distributed attack sources of the non-periodic aggregated distributed LDoS attack pulse generate non-uniform and non-periodic waveforms with the same burst size, but the converged attack flow is periodic. To avoid frequency domain detection, the distributed LDoS attack flow must meet the following requirements: (a) the aggregated flow is a traditional LDoS flow. (b) the distributed traffic has a frequency distribution similar to normal traffic. This application's embodiment selects a one-dimensional random walk algorithm to generate the distributed flow. The one-dimensional random walk problem refers to a point mass (random walker) moving along a straight line, moving only one unit length per unit time, and only stopping at integer points on the line. Assume that at time t, the point mass is located at point i on the line. p Then at time t+1, the position of the particle has three possibilities: ① It jumps to the integer point i with probability p. p -1② or jump to point i with probability q. p +1③ or remain at point i with probability r = 1 - pq. p Since the outcome of each step is independent and the sum of the probabilities of each outcome is 1, the process follows a Bernoulli distribution and is called a Bernoulli random walk process.

[0107] DLDoS based on one-dimensional walk algorithm can be defined by a seven-tuple. <N a ,R',T a ,T b ,L',M',S'>。 Wherein, N a Here, R is the attack node, R' is the rate of each attack pulse, L' is the duration of each attack pulse, and T is the attack node. a T is the interval between attack cycles. b M' is the interval between each pulse in the attack cycle, M' is the number of pulses in the attack cycle, and S' is the number of computers controlled by the attacking node.

[0108] The goal of an LDOS attack is to prevent measurement data from being transmitted to the monitoring center, causing data delays or gaps. This prevents the monitoring center from accurately estimating the system status, making incorrect scheduling decisions, and ultimately overloading generators or transmission lines. We introduce an availability attack vector d∈{0,1}. m For LDoS attacks, d(j) = 1 indicates that the measurement value j is unavailable. Therefore, the model of the remaining measurement value and system state can be represented by z. d =H d x+e d , where e d It is a noise vector, z d H is the vector of remaining measurements. d ∈R m ×nThe system matrix representing the remaining measurements is obtained from H by replacing some rows with zero row vectors, because these measurements are subject to LDoS attacks, i.e., H. d =(I G -diag(d))H. The mathematical model for an LDOS attack is as follows:

[0109] max D a ldos =w g ldos D g ldos +w b ldos D b ldos (35)

[0110]

[0111]

[0112] In the formula: D a ldos Represents the effect of an LDoS attack; D g ldos and D b ldos These represent the number of generator overloads and transmission line overloads caused by an LDoS attack; r ldos For the measurement value matrix uploaded to the monitoring center, This is a matrix of measurement values ​​after being subjected to an LDoS attack.

[0113] The embodiments of this application can model the partially observable Markov decision process of a multi-agent system as a tuple, and then use the reward function and other elements in the tuple to control the optimization of the cooperative attack strategy. This can effectively promote cooperation among agents and help agents focus on their respective tasks, making the final cooperative network attack model more comprehensive, accurate and effective.

[0114] Optionally, in one embodiment of this application, it further includes: generating a communication-based reinforcement learning Gated-BiCNet algorithm for optimizing cooperative network attack strategies through centralized optimal policy learning and distributed execution strategies under interactive communication, so as to optimize the effective data ratio of communication information between agents using the communication-based reinforcement learning Gated-BiCNet algorithm.

[0115] Among them, centralized optimal strategy learning and distributed execution strategy can be understood as a hybrid architecture that combines the advantages of centralized training with the efficiency of distributed execution.

[0116] This application embodiment can generate the Gated-BiCNet algorithm based on at least a Gated-Actor network and at least one Critic network, utilizing the aforementioned centralized optimal policy learning and distributed execution strategy under interactive communication. The structure of the Gated-BiCNet algorithm is as follows: Figure 2 As shown.

[0117] The embodiments of this application can optimize the communication process of cooperative network attack strategies based on the generated communication-type reinforcement learning Gated-BiCNet algorithm, thereby improving attack strength and efficiency.

[0118] Optionally, in one embodiment of this application, the Gated-BiCNet algorithm includes at least one Gated-Actor network and at least one Critic network. Each Gated-Actor network and each Critic network belongs to an agent. Each Gated-Actor network and each Critic network is implemented based on a bidirectional recurrent neural network. Each agent in the Gated-BiCNet inputs observation information into a gate, which filters redundant data from the input information. Each Gated-Actor network generates and executes an action based on the removed redundant information. Each Critic network evaluates the action's Q-value based on the observation information and then updates the network parameters.

[0119] In actual implementation, the gating mechanism of this application embodiment can generate the validity probability corresponding to the observation information and determine whether to encourage communication or perform message pruning based on the value of the validity probability, thereby reducing redundant information.

[0120] The embodiments of this application can prune redundant communication information by setting a gating filtering mechanism, thereby increasing the proportion of effective data in communication information between agents, thus improving communication efficiency and enhancing the cooperation and attack effect between agents in the process of cooperative network attack.

[0121] Optionally, in one embodiment of this application, the effective data ratio of communication information between agents is optimized using the communication-based reinforcement learning Gated-BiCNet algorithm, including: generating the effectiveness probability corresponding to the observation information; training the effectiveness probability using at least one preset auxiliary task, so that if the communication message is valid, it is input into the Actor network, otherwise it is determined that there is redundant data in the communication message, so that the information after multiplying the vector of the communication message with the zero vector is input into the Actor network.

[0122] like Figure 3 As shown, the redundant information filtering steps of the gating mechanism are as follows:

[0123] Step A1: Generate the validity probability p (i.e., the value ΔQ(o)) corresponding to the observation information. i () greater than the pruning threshold T d The probability of ( ) is shown in formula (47):

[0124]

[0125] In the formula, Alone is the probability indicator function; Y(o i ) represents the label for auxiliary tasks, with a value range of {0,1}; Actions generated by agent i based on communication information and local information; The action generated by agent i based on local information; and They are respectively and The corresponding policy evaluation value (Q-value) is generated by the Critic network.

[0126] Step A2: Train the effectiveness probability p using an auxiliary task, and minimize the loss function L(o i Training is performed as shown in formula (48).

[0127]

[0128] Step A3: When p = Y(o i When the value of ) is 1, In other words, at this point, a value greater than the pruning threshold T should be generated. d The probability p is used to encourage communication because the Q-value of actions generated based on communication messages is larger; when p = Y(o i When the value of ) is 0, In other words, if the Q-value of an action generated based on communication messages and local messages is less than the Q-value of an action generated based on local messages, then message pruning is performed to reduce redundant information.

[0129] Step A4: If p = Y(o i If the value of ) is 1, the communication message is considered valid and is input into the Actor network; if p = Y(o i If the value of ) is 1, it is considered that there is redundant data in the communication message, and the information after multiplying the communication message vector and the zero vector is input into the Actor network.

[0130] This application embodiment can compare the Q-value of an action generated based on a communication message with the Q-value of an action generated based on a local message by judging the value of the validity probability, thereby determining whether message pruning is needed for the communication message, ensuring that valid communication information is input into the Actor network, filtering redundant information, and increasing the proportion of valid communication information between agents.

[0131] Optionally, in one embodiment of this application, the training formula for the validity probability is:

[0132]

[0133] Among them, L(o i Y(o) is the loss function to be minimized. i ) is the label for auxiliary tasks, p(o i ) represents the gating probability, o i For the observation of agent i, The expectation symbol represents the expectation of o. i The expected value is calculated from the probability distribution.

[0134] In actual implementation, the embodiments of this application can use parameters such as auxiliary task labels to construct a training model of effectiveness probability, and train the effectiveness probability model with minimizing loss as the training objective.

[0135] The embodiments of this application can optimize the training of the validity probability model by minimizing the loss function, effectively improving its prediction accuracy, thereby ensuring the input of valid communication messages and filtering redundant and invalid information.

[0136] The power system cooperative network attack modeling method proposed in this application can model the covert attack behavior of spatiotemporal coordination of various attack methods as a multi-agent partially observable Markov decision process problem, thereby constructing a cooperative network attack model. This model can comprehensively characterize various possible covert cooperative network attack behaviors under partially observable system state conditions. Furthermore, a gated communication-based reinforcement learning algorithm, Gated-BiCNet, is designed to optimize the cooperative network attack strategy. By pruning redundant communication information, the effective data ratio of communication information between agents is increased, enhancing the cooperation and attack effectiveness among agents during the cooperative network attack process. This solves the problems in related technologies where attack models cannot effectively characterize various attack behaviors and do not consider the impact of attack strategy optimization and information sharing during execution on attack cooperation and effectiveness improvement, resulting in unreliable attack effectiveness.

[0137] Next, refer to the appendix. Figure 4 This application describes a power system cooperative network attack modeling apparatus according to an embodiment.

[0138] Figure 4 This is a block diagram of a power system cooperative network attack modeling device according to an embodiment of this application.

[0139] like Figure 4 As shown, the power system collaborative network attack modeling device 10 includes a first modeling module 100, a generation module 200, and a second modeling module 300.

[0140] The first modeling module 100 is used to model the covert attack behavior of multiple attack methods in a spatiotemporal coordination under partially observable conditions of a power system as a multi-agent partially observable Markov decision process problem.

[0141] The generation module 200 is used to generate tuples of power systems based on multi-agent partially observable Markov decision process problems.

[0142] The second modeling module 300 is used to construct a hidden cooperative network attack model based on the power system's state partial observability conditions using a tuple. The tuple includes at least one of the following: agent set, system state space, joint action space, state transition probability, reward function, joint observation space, observation probability function, and discount factor.

[0143] Optionally, in one embodiment of this application, the power system cooperative network attack modeling device 10 further includes an optimization module, which is used to generate a communication-based reinforcement learning Gated-BiCNet algorithm for cooperative network attack strategy optimization through centralized optimal strategy learning and distributed execution strategy under interactive communication, so as to optimize the effective data ratio of communication information between agents using the communication-based reinforcement learning Gated-BiCNet algorithm.

[0144] Optionally, in one embodiment of this application, the Gated-BiCNet algorithm includes at least one Gated-Actor network and at least one Critic network, each Gated-Actor network and each Critic network belongs to an agent, each Gated-Actor network and each Critic network is implemented based on a bidirectional recurrent neural network, and each agent in the Gated-BiCNet inputs observation information into a gate, the gate filters redundant data of the input information, each Gated-Actor network generates and executes an action based on the removed redundant information, and each Critic network evaluates the value Q-value of the action based on the observation information and then updates the network parameters.

[0145] Optionally, in one embodiment of this application, the optimization module includes a generation unit and an input unit; wherein, the generation unit is used to generate the validity probability corresponding to the observation information; the input unit is used to train the validity probability using at least one preset auxiliary task, so that if the communication message is valid, it is input into the Actor network, otherwise it is determined that there is redundant data in the communication message, so that the information after multiplying the vector of the communication message with the zero vector is input into the Actor network.

[0146] Optionally, in one embodiment of this application, the training formula for the validity probability is:

[0147]

[0148] Among them, L(o i Y(o) is the loss function to be minimized. i ) is the label for auxiliary tasks, p(o i ) represents the gating probability, o i For the observation of agent i, The expectation symbol represents the expectation of o. i The expected value is calculated from the probability distribution.

[0149] It should be noted that the foregoing explanation of the embodiment of the power system collaborative network attack modeling method also applies to the power system collaborative network attack modeling device of this embodiment, and will not be repeated here.

[0150] The power system cooperative network attack modeling device proposed in this application can model the covert attack behavior of spatiotemporal cooperation among various attack methods as a multi-agent partially observable Markov decision process problem, thereby constructing a cooperative network attack model. This model can comprehensively characterize various possible covert cooperative network attack behaviors under partially observable system state conditions. Furthermore, a gated communication-based reinforcement learning algorithm, Gated-BiCNet, is designed to optimize the cooperative network attack strategy. By pruning redundant communication information, the effective data ratio of communication information between agents is increased, enhancing the cooperation and attack effectiveness among agents during the cooperative network attack process. This solves the problems in related technologies where attack models cannot effectively characterize various attack behaviors and do not consider the impact of attack strategy optimization and information sharing during execution on attack cooperation and effectiveness improvement, resulting in unreliable attack effectiveness.

[0151] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0152] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0153] When processor 502 executes the program, it implements the power system cooperative network attack modeling method provided in the above embodiments.

[0154] Furthermore, electronic devices also include:

[0155] Communication interface 503 is used for communication between memory 501 and processor 502.

[0156] The memory 501 is used to store computer programs that can run on the processor 502.

[0157] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0158] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0159] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0160] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0161] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described power system cooperative network attack modeling method.

[0162] This application also provides a computer program product on which a computer program is stored, which, when executed by a processor, implements the above-described power system cooperative network attack modeling method.

[0163] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0164] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0165] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0166] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0167] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0168] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0169] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0170] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for modeling cooperative network attacks in power systems, characterized in that, Includes the following steps: Based on the spatiotemporal coordination of multiple attack methods under partially observable conditions of power systems, the stealth attack behavior of multiple attack methods is modeled as a multi-agent partially observable Markov decision process problem. Based on the multi-agent partially observable Markov decision process problem, a tuple of the power system is generated. Based on the power system, a cooperative network attack model is constructed under the condition that the system state is partially observable. The tuple includes at least one of the following: agent set, system state space, joint action space, state transition probability, reward function, joint observation space, observation probability function, and discount factor.

2. The method according to claim 1, characterized in that, Also includes: By using centralized optimal policy learning and distributed execution strategies under interactive communication, a communication-based reinforcement learning algorithm, Gated-BiCNet, is generated for optimizing cooperative network attack strategies. This algorithm is then used to optimize the effective data ratio of communication information between agents.

3. The method according to claim 2, characterized in that, The Gated-BiCNet algorithm includes at least one Gated-Actor network and at least one Critic network. Each Gated-Actor network and each Critic network belongs to an agent. Each Gated-Actor network and each Critic network are implemented based on a bidirectional recurrent neural network. Each agent in the Gated-BiCNet inputs observation information into a gate, which filters redundant data from the input information. Each Gated-Actor network generates and executes an action based on the removed redundant information. Each Critic network evaluates the action's Q-value based on the observation information and then updates the network parameters.

4. The method according to claim 3, characterized in that, The optimization of the effective data ratio of communication information between agents using the Gated-BiCNet algorithm for communication-based reinforcement learning includes: Generate the validity probability corresponding to the observation information; The validity probability is trained using at least one preset auxiliary task, so that if the communication message is valid, it is input into the Actor network; otherwise, it is determined that there is redundant data in the communication message, so that the information obtained by multiplying the vector of the communication message with the zero vector is input into the Actor network.

5. The method according to claim 4, characterized in that, The training formula for the validity probability is: Among them, L(o i Y(o) is the loss function to be minimized. i ) is the label for auxiliary tasks, p(o i ) represents the gating probability, o i For the observation of agent i, The expectation symbol represents the expectation of o. i The expected value is calculated from the probability distribution.

6. A power system cooperative network attack modeling device, characterized in that, include: The first modeling module is used to model the spatiotemporal coordinated covert attack behavior of multiple attack methods under partially observable conditions of the power system as a multi-agent partially observable Markov decision process problem. The generation module is used to generate a tuple of the power system based on the multi-agent partially observable Markov decision process problem. The second modeling module is used to construct a hidden cooperative network attack model under the condition that the system state is partially observable based on the tuple of the power system. The tuple includes at least one of the following: agent set, system state space, joint action space, state transition probability, reward function, joint observation space, observation probability function, and discount factor.

7. The apparatus according to claim 6, characterized in that, Also includes: The optimization module is used to generate a communication-based reinforcement learning algorithm, Gated-BiCNet, for optimizing cooperative network attack strategies through centralized optimal strategy learning and distributed execution strategies under interactive communication. This algorithm is used to optimize the effective data ratio of communication information between agents.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the power system cooperative network attack modeling method as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the power system cooperative network attack modeling method as described in any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the power system cooperative network attack modeling method as described in any one of claims 1-5.

Images