Offline reinforcement learning method based on multi-round FDIA path generation

Abstract

The invention discloses an off-line reinforcement learning method based on multi-round FDIA path generation, and belongs to the technical field of power system information security and partial observable off-line reinforcement learning intersection, and the method comprises the following steps: S1, collecting an off-line data set and initializing a function module; s2, generating a real transfer sample; s3, generating a virtual transfer sample; s4, mixed data set training and network updating; and S5, multi-round attack path generation and simulation verification. According to the offline reinforcement learning method based on multi-round FDIA path generation, a complete attack path across a scheduling period is generated, dynamic risk accumulation is accurately described, offline training is stable and reliable, online power grid tentative detection is not needed, safety risks are avoided, the system cascade instability risk can be accurately evaluated, a real attack scene is fit, and the method is suitable for large-scale popularization and application. And the implementability in an actual power grid is obviously improved.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary technical field of power system information security and partially observable offline reinforcement learning, specifically involving an offline reinforcement learning method based on multi-round FDIA path generation. Background Technology

[0002] As the core of energy supply, the power system has the inherent characteristics of dynamic evolution and continuous iterative updates of its state over multiple scheduling cycles: the grid load fluctuates over time, the unit output is adjusted in real time, and the topology may change due to faults or maintenance. If an attacker wants to achieve the destructive goal of inducing cascading instability or even a major blackout, they must gradually amplify the risk through continuous attacks over multiple scheduling cycles, rather than simply injecting false data once.

[0003] However, existing technologies have the following shortcomings: they only design single-round attack vectors, which cannot describe the complete evolution process of "initial disturbance, cascade instability, and power outage"; online reinforcement learning has high interaction costs and security risks, while offline reinforcement learning is prone to distribution shift and value overestimation, leading to policy instability; they only focus on static indicators such as voltage overruns and line overloads, and the frequency is only used for post-event analysis, without being incorporated into the design of attack targets and rewards; they ignore the system state changes in adjacent scheduling cycles and the fact that attackers cannot continuously intervene, and do not combine real-time and post-attack state evolution planning paths.

[0004] Therefore, a new method is urgently needed.

[0005] The technical means related to the attack and defense game of the power system are only used for the construction and verification of the power network's own security protection system. They are intended to identify and resist external malicious attacks in advance and ensure the stable operation of the power system. They are by no means intended for malicious intrusion and damage to power facilities or the power grid. Summary of the Invention

[0006] The purpose of this invention is to provide an offline reinforcement learning method based on multi-round FDIA path generation. This method generates complete attack paths across scheduling cycles, accurately characterizes dynamic risk accumulation, and provides stable and reliable offline training without requiring online grid testing and posing no security risks. It can accurately assess the risk of system cascading instability, closely matches real attack scenarios, and significantly improves its feasibility in actual power grids.

[0007] To achieve the above objectives, this invention provides an offline reinforcement learning method based on multi-round FDIA path generation, comprising the following steps: S1. Collect offline data from multiple scheduling cycles of the power system as the basic data for the entire process, and initialize the modules and parameters; S2 receives the offline data and initialization parameters collected by S1, extracts offline data in batches, maps local observations to global belief states through the belief state estimator, generates attack actions in combination with the policy network, calculates the real reward according to the frequency-voltage joint rule, and forms real transfer samples. S3 receives the true belief state transmitted by S2, the modules and parameters initialized by S1, converts the true belief state into a graph structure input containing the power grid topology, predicts the next model belief state through an attention mechanism, and generates a sequence in a loop to form virtual transition samples. S4. Receive the real transfer samples from S2, the virtual transfer samples from S3, and the modules and parameters initialized in S1. Merge the two types of samples to construct a mixed dataset. Use a pruned QR learning method to update the online value network and the target value network. Update the policy network using a policy gradient with entropy regularization. Obtain the trained belief state estimator, policy network, and value network parameters. S5 receives the online observations from the power system in real time, including the parameters of the belief state estimator, policy network, and value network trained by S4; maps the online observations to the current belief state through the belief state estimator, outputs attack actions through the policy network, and generates attack paths across scheduling cycles and evaluation reports.

[0008] Preferably, in S1, the offline data includes observations, the actual system state, attack actions, and reward information; the initialization of core modules and parameters includes a belief state estimator, a state prediction module, a policy network, an online value network, a target value network, and entropy-temperature parameters and hyperparameters; the hyperparameters include the number of rolling steps; the policy network is an Actor network, and the target value network includes Target... Sub-network and Target Sub-networks, the online value network includes Subnetworks and Subnetwork.

[0009] Preferably, in S2, the real transfer sample is a sequence structure of "real belief state - real attack action - real reward - next real belief state".

[0010] Preferably, in S2, the local observations are mapped to the global belief state through the belief state estimator, specifically as follows: Local observations from the power system output are received and input into the belief state estimator to resolve state confusion issues, thus mapping the true belief states. ; The calculation of the actual reward based on the frequency-voltage joint rule is as follows: A joint frequency-voltage reward function is constructed based on the frequency offset and voltage over-limit conditions of the power system. The actual reward is then calculated by substituting the system response data corresponding to the attack action. .

[0011] Preferably, S3 includes the following steps: S301. Take the true belief state sample transmitted in S2, with power grid components as nodes and the connections between components as edges. The node feature vector contains the time. The component's running state is converted into a graph structure input adapted by Graphormer; S302. Based on the graph structure of S301, the attention weights between nodes are recalculated through an attention mechanism, using the following formula: ; In the formula, For nodes At any moment eigenvectors, This is the weight matrix. For bias, For feature dimensions; S303. Based on the attention weights calculated by S302, the fixed graph topology and the current node running state information are fused to predict the next model belief state. S304. The policy network takes the model belief state output by S303 as input and generates an attack action that adapts to the current state; it calculates the model reward corresponding to the attack action according to the frequency-voltage joint rule. S305, Execute S302-S305 in a loop, store the sequence of "current model belief state - attack action - model reward - next model belief state", and form a virtual transition sample; The loop will terminate when any of the following conditions are met: The loop count reaches the preset number of scrolling steps initialized in S1; The difference between the model belief states in two consecutive outputs is less than a preset threshold; The total number of virtual transfer samples generated meets the training requirements of the mixed dataset.

[0012] Preferably, in S4, a trimmed QR learning method is used to update the online value network and the target value network, specifically as follows: Computing online value networks , For the current state-action output value, take the smaller of the two values ​​as the current value estimate; calculate the target value network Target. Target For the output value of the next state, the smaller of the two values ​​is taken as the target value benchmark; a loss function is constructed by constraining the range of the difference between the current value estimate and the target value benchmark, the online value network is updated, and the target value network parameters are updated at a fixed frequency.

[0013] Therefore, the present invention employs the above-mentioned offline reinforcement learning method based on multi-round FDIA path generation. Compared with the prior art, the technical solution of the present invention has the following beneficial effects: (1) By adopting the partially observable Markov decision process modeling technique, the technical problem of "the limitation of single-round static modeling, which cannot depict the dynamic risk accumulation and complete attack path across scheduling cycles" is overcome, thereby achieving the technical effect of "generating complete attack paths across scheduling cycles and accurately depicting the dynamic risk accumulation process". (2) By using the technique of generating virtual samples with Graphormer and constructing a "real model" hybrid dataset, the technical problems of "distribution shift, overestimation of value, and security risks and high costs of online power grid testing in offline deployment of reinforcement learning" are overcome, thereby achieving the technical effect of "obtaining a stable and reliable offline training attack strategy without online power grid testing"; (3) By incorporating frequency offset / evolution as a core state indicator into the reward design, the technical problem of "not making full use of frequency dynamic indicators and relying solely on static indicators to make it difficult to assess the risk of system cascade instability" is overcome, thereby achieving the technical effect of "accurately assessing the risk of system cascade instability and making up for the lack of static indicators". (4) By using a belief state estimator to map local measurements to global states and considering the changes in system state within the scheduling cycle, the technical problem of “incomplete attacker information and inability to continuously intervene in real attack scenarios, and insufficient fit of existing solutions” is overcome, thereby achieving the technical effect of “fitting real attack scenarios and improving the feasibility of the technology in real power grids”.

[0014] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0015] Figure 1 This is an architecture diagram of an embodiment of the offline reinforcement learning method based on multi-round FDIA path generation according to the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used in the present invention should have the ordinary meaning understood by those skilled in the art.

[0017] Example 1 like Figure 1 As shown, this embodiment provides an offline reinforcement learning method based on multi-round FDIA path generation. It should be understood that the specific parameters, models and protocols mentioned in this embodiment are merely examples to help those skilled in the art understand the present invention, and are not intended to limit the present invention.

[0018] The offline reinforcement learning method based on multi-round FDIA path generation of the present invention includes the following steps: S1. Collect offline datasets from multiple scheduling cycles of the power system and initialize core modules and parameters; In this step, the offline data includes observations, the actual state of the system, attack actions, rewards, and other information. Initialize the core modules and parameters, including the belief state estimator, the Graphormer state prediction module with an attention mechanism, the policy network (Actor), the policy network, the online value network, the target value network, and hyperparameters such as entropy temperature parameter and rolling steps; S2 receives offline data collected by S1 and initializes parameters; batch extracts offline data, maps local observations to global belief states through a belief state estimator, generates attack actions in combination with a policy network, calculates the real reward according to the frequency-voltage joint rule, and forms real transfer samples. In this step, the power system outputs local observations, which are then input to the belief state estimator to resolve the state confusion problem and map them to the true belief states. ; Mini-batch samples are extracted from the offline dataset T and combined with a policy network to generate realistic attack actions. Calculate the true reward using the frequency-voltage joint reward function. ; Construct realistic transition samples of "real belief state - real attack action - real reward - next real belief state"; S3 receives the real belief state samples, S1 initialization module and parameters transmitted by S2; converts the real belief state into a graph structure input containing the power grid topology, predicts the next model belief state through the Graphormer attention mechanism, and cyclically generates the "model belief state-attack action-model reward" sequence to alleviate the distribution offset problem. S301. Take the true belief state sample transmitted in S2, with power grid components (generators, lines, loads, etc.) as nodes and the connections between components as edges. The node feature vector contains the time... The component's running state is converted into a graph structure input adapted by Graphormer; S302. Based on the graph structure of S301, the attention weights between nodes are recalculated through an attention mechanism, using the following formula: ; In the formula, For nodes At any moment eigenvectors, This is the weight matrix. For bias, For feature dimensions; S303, based on the attention weights calculated by S302, integrates the fixed graph topology and the current node running state information to predict the next model belief state, thus simultaneously alleviating the "distribution offset problem" of offline reinforcement learning; S304. The policy network takes the model belief state output by S303 as input and generates an attack action that adapts to the current state; it calculates the model reward corresponding to the attack action according to the frequency-voltage joint rule. S305, Execute S302-S305 in a loop, store the sequence of "current model belief state - attack action - model reward - next model belief state", and form a virtual transition sample; The loop will terminate when any of the following conditions are met: The loop count reaches the preset number of scrolling steps initialized in S1; The difference between the model belief states in two consecutive outputs is less than a preset threshold; The total number of virtual transfer samples generated meets the training requirements of the mixed dataset.

[0019] S4: Receive the real transfer samples from S2, the virtual transfer samples from S3, and the initialization module and parameters from S1; merge the real transfer samples and the virtual transfer samples to construct a hybrid dataset, and use the hybrid dataset to train the policy network and the value network; obtain the trained belief state estimator, policy network, and value network parameters. In this step, real transition samples and virtual transition samples are merged to construct a hybrid dataset containing "belief state - attack action - reward - next belief state"; Training using a mixed dataset: The policy network (Actor) mitigates the "risk of policy overestimation" by updating policy gradients with entropy regularization. Online value network (Critic) includes , Two sub-networks, the Target Value Network (Target Q) includes Target Target Two subnetworks; online computation , For the output value of the current state and action, take the smaller value as the current value estimate to alleviate overestimation of online value; calculate the Target. Target For the output value of the next state, take the smaller of the two values ​​as the target value benchmark; A pruned QR learning approach is adopted. By constraining the range of the difference between the current value estimate and the target value benchmark, a loss function is constructed to update the online value network. The parameters of the target value network are updated synchronously at a fixed frequency, and a conservative term is introduced to further suppress value overestimation. S5 receives the online observations from the power system in real time, including the parameters of the belief state estimator, policy network, and value network trained by S4; maps the online observations to the current belief state through the belief state estimator, outputs attack actions through the policy network, and generates attack paths across scheduling cycles and evaluation reports. S501, a fixed-trained belief state estimator and policy network, establish a real-time data interaction link with the power system, receive online observations of the current scheduling cycle, and map them to the global belief state through the belief state estimator. ; S502, with As input, simulated attack actions are output through the policy network. After being input into the power system, online observations for the next scheduling cycle are obtained. The "state mapping-action generation-feedback acquisition" process is executed cyclically until the preset total number of scheduling cycles is reached or the instability warning threshold is triggered, thus forming a complete attack path across scheduling cycles. S503: Simulate and reproduce the dynamic response of the power grid under the attack path, evaluate the core security indicators such as system frequency offset, voltage over-limit duration, line overload and cascade instability trigger probability, and determine whether the attack effect meets the standard. S504. Compile two types of results: one is a complete attack path containing attack actions and state evolution data for each cycle; the other is a validity assessment report containing simulation results of security indicators and conclusions on attack effects.

[0020] Therefore, this invention adopts the above-mentioned offline reinforcement learning method based on multi-round FDIA path generation. This method generates complete attack paths across scheduling cycles, accurately depicts dynamic risk accumulation, and provides stable and reliable offline training without the need for online grid testing and without security risks. It can accurately assess the risk of system cascading instability, closely matches real attack scenarios, and significantly improves its feasibility in actual power grids.

[0021] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0022] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. An offline reinforcement learning method based on multi-round FDIA path generation, characterized in that, Includes the following steps: S1. Collect offline data from multiple scheduling cycles of the power system as the basic data for the entire process, and initialize the modules and parameters; S2 receives the offline data and initialization parameters collected by S1, extracts offline data in batches, maps local observations to global belief states through the belief state estimator, generates attack actions in combination with the policy network, calculates the real reward according to the frequency-voltage joint rule, and forms real transfer samples. S3 receives the true belief state transmitted by S2, the modules and parameters initialized by S1, converts the true belief state into a graph structure input containing the power grid topology, predicts the next model belief state through an attention mechanism, and generates a sequence in a loop to form virtual transition samples. S4 receives the real transfer samples from S2, the virtual transfer samples from S3, and the modules and parameters initialized in S1. It merges the two types of samples to construct a mixed dataset, uses a pruned QR learning method to update the online value network and the target value network, and updates the policy network through a policy gradient with entropy regularization. It obtains the trained belief state estimator, policy network, and value network parameters. S5 receives the online observations from the power system in real time, including the parameters of the belief state estimator, policy network, and value network trained by S4; maps the online observations to the current belief state through the belief state estimator, outputs attack actions through the policy network, and generates attack paths across scheduling cycles and evaluation reports.

2. The offline reinforcement learning method based on multi-round FDIA path generation according to claim 1, characterized in that, In S1, the offline data includes observations, the actual system state, attack actions, and reward information; the core modules and parameters are initialized, including a belief state estimator, a state prediction module, a policy network, an online value network, a target value network, and entropy-temperature parameters and hyperparameters; the hyperparameters include the number of rolling steps; the policy network is an Actor network, and the target value network includes Target... Sub-network and Target Sub-networks, the online value network includes Subnetwork and Subnetwork.

3. The offline reinforcement learning method based on multi-round FDIA path generation according to claim 1, characterized in that, In S2, the real transfer sample is a sequence structure of "real belief state - real attack action - real reward - next real belief state".

4. The offline reinforcement learning method based on multi-round FDIA path generation according to claim 1, characterized in that, In S2, the local observations are mapped to the global belief state through the belief state estimator, specifically as follows: Local observations from the power system output are received and input into the belief state estimator to resolve state confusion issues, thus mapping the true belief states. ; The calculation of the actual reward based on the frequency-voltage joint rule is as follows: A joint frequency-voltage reward function is constructed based on the frequency offset and voltage over-limit conditions of the power system. The actual reward is then calculated by substituting the system response data corresponding to the attack action. .

5. The offline reinforcement learning method based on multi-round FDIA path generation according to claim 1, characterized in that, S3 includes the following steps: S301. Take the true belief state sample transmitted in S2, with power grid components as nodes and the connections between components as edges. The node feature vector contains the time. The component operating status is converted into an adapted graph structure input; S302. Based on the graph structure of S301, the attention weights between nodes are recalculated through an attention mechanism, using the following formula: ； In the formula, For nodes At any moment eigenvectors, This is the weight matrix. For bias, For feature dimensions; S303. Based on the attention weights calculated by S302, the fixed graph topology and the current node running state information are fused to predict the next model belief state. S304. The policy network takes the model belief state output by S303 as input and generates an attack action that adapts to the current state; it calculates the model reward corresponding to the attack action according to the frequency-voltage joint rule. S305, Execute S302-S305 in a loop, store the sequence of "current model belief state - attack action - model reward - next model belief state" to form a virtual transition sample; The loop will terminate when any of the following conditions are met: The loop count reaches the preset number of scrolling steps initialized in S1; The difference between the model belief states in two consecutive outputs is less than a preset threshold; The total number of virtual transfer samples generated meets the training requirements of the mixed dataset.

6. The offline reinforcement learning method based on multi-round FDIA path generation according to claim 1, characterized in that, In S4, a modified QR learning method is used to update the online value network and the target value network, specifically as follows: Computing online value networks , For the current state-action output value, take the smaller of the two values ​​as the current value estimate; calculate the target value network Target. Target For the output value of the next state, the smaller of the two values ​​is taken as the target value benchmark; a loss function is constructed by constraining the range of the difference between the current value estimate and the target value benchmark, the online value network is updated, and the target value network parameters are updated at a fixed frequency.

7. A computer device, characterized in that, include: A processor configured to be coupled to memory, read and execute instructions and / or program code in the memory to perform the method as described in any one of claims 1-6.

8. A computer-readable medium, characterized in that, The computer-readable medium stores computer program code that, when executed on a computer, causes the computer to perform the method as described in any one of claims 1-6.

Images