A load frequency control system attack detection method based on reinforcement learning

Abstract

The present invention belongs to the field of power system security technology and discloses a load frequency control system attack detection method based on reinforcement learning. The method aims to enhance the power system's ability to identify and defend against complex network attacks, and address the shortcomings of traditional detection methods in attack sample generation, unknown attack identification, and system adaptability. The present invention constructs an attack agent based on a Markov decision process and uses an improved reinforcement learning algorithm to generate highly concealed adversarial samples. It designs a dual-modal detection architecture that integrates LSTM supervised learning and autoencoder unsupervised learning, and introduces an adaptive weight fusion mechanism to achieve attack type identification and anomaly detection. Incremental learning is combined with a dynamic parameter adjustment mechanism to enable the detection model to have continuous learning and evolution capabilities. This method can be applied to power grid dispatching centers or smart microgrids to achieve real-time monitoring and attack defense of load frequency control systems, significantly improving the operational safety and robustness of power systems.

Description

Technical Field

[0001] The present invention belongs to the technical field of power system security, and in particular relates to a load frequency control system attack detection method based on reinforcement learning. Background Art

[0002] Load frequency control (LFC) system security detection technology is a key branch of power system cybersecurity, focusing on how to effectively identify and defend against various cyberattacks targeting power system frequency regulation mechanisms. This technology integrates knowledge from multiple disciplines, including power system control theory, cybersecurity, artificial intelligence, and data analysis, aiming to ensure the stable operation, security, and reliability of power systems. In recent years, with the increasing digitalization and intelligence of power systems, the cybersecurity threats facing power infrastructure have increased. In particular, attack vectors targeting LFC systems have evolved, from simple denial-of-service attacks to more complex advanced persistent threats such as false data injection and perturbation attacks. Traditional rule- and signature-based detection methods are unable to combat these highly concealed and adaptable attack vectors. Intelligent detection technologies based on machine learning, particularly reinforcement learning, have gradually become a research hotspot, demonstrating enhanced attack identification capabilities and adaptability. The significance of researching LFC attack detection technology lies in the fact that effective attack detection mechanisms are critical to ensuring the safe and stable operation of power systems. On the one hand, accurate attack detection can promptly detect malicious activity targeting the LFC system, preventing widespread power outages and equipment damage caused by grid frequency outages. On the other hand, a detection system with adaptive learning capabilities can continuously respond to evolving attack vectors, ensuring the long-term safe operation of the power system. With the rapid development of the Energy Internet and the advancement of smart grid construction, establishing efficient and reliable LFC system attack detection methods is of great practical significance for maintaining national energy security and promoting the healthy development of the power industry.

[0003] Currently, relevant technologies focus on the following three aspects: first, security enhancement and defense strategies in power grid operation and control, such as reinforcement learning control methods and game defense mechanisms; second, data-driven anomaly detection and attack identification technologies, including multimodal detection, time series prediction, state estimation and other methods; third, intelligent assessment and risk analysis of power system structure and operating status, such as three-dimensional modeling, power flow analysis and system vulnerability assessment.

[0004] In the field of attack detection for power system LFC systems, existing technologies have the following deficiencies:

[0005] 1) Attack sample generation methods lack diversity and realism, making it difficult to cover complex attack scenarios. Existing methods primarily rely on reproducing historical data or random perturbations to generate attack samples, failing to fully simulate highly concealed and high-impact adversarial attacks. The lack of systematic design of attack samples results in insufficient attack diversity during detection model training, making it difficult to cope with complex attack patterns that pose real threats.

[0006] 2) Detection models have weak recognition capabilities for unknown attack types and insufficient generalization capabilities. Most existing technologies use only supervised learning methods for attack classification and detection, relying on known attack samples for training. This makes it difficult to identify attack types that do not appear in the training set, resulting in a significant generalization bottleneck.

[0007] 3) Detection methods lack the ability to continuously learn and adapt, making them incapable of responding to evolving attacks over the long term. Existing detection systems often deploy static models with fixed parameters and thresholds. These models are unable to adapt to new attack patterns or changes in system operating status, resulting in a decline in detection performance over time. Summary of the Invention

[0008] To solve the above technical problems, the present invention provides a load frequency control system attack detection method based on reinforcement learning. On the basis of improving the authenticity and diversity of attack samples, it effectively enhances the detection model's ability to identify unknown attacks and the long-term stability and adaptability of the system, and significantly improves the security protection level of the load frequency control system in the face of complex network attacks.

[0009] The load frequency control system attack detection method based on reinforcement learning described in the present invention includes the following steps:

[0010] Step 1: Based on the key state variables and parameters of the load frequency control system, with the goal of building a complete data set, a standardized training set, validation set, and test set are constructed through multi-stage data collection and processing;

[0011] Step 2: Based on the Markov decision process theory framework, with the goal of generating efficient and covert attack samples, we train the attack agent by constructing a multi-objective reward function and an improved REINFORCE algorithm to construct a set of false data injection and load switching attack samples;

[0012] Step 3: Based on time series features and system status data, with the goal of achieving accurate and reliable attack detection, an attack detection model is constructed that includes a supervised learning detection model, an unsupervised learning model, and a fusion decision mechanism.

[0013] Step 4: Based on the trained attack detection model, build an attack detection and defense system with the goal of optimizing detection performance and system adaptability.

[0014] Furthermore, step 1 is specifically as follows:

[0015] The key state variables of the load frequency control system are collected at a preset sampling frequency to obtain the system state vector ,in, is the frequency integration error, is the speed regulator power deviation, is the mechanical power deviation, is the system frequency deviation, is the transpose of the vector; at the same time, the control input is collected and load disturbances , and key system parameters ; The key parameters of the system Including governor time constant , turbine time constant , system inertia constant , damping coefficient and the governor droop coefficient , are static values;

[0016] For all sampled dynamic variables, including system state variables 、 、 、 , and system control input and load disturbances , Z-score standardization method was applied for processing;

[0017] Clean the data after Z-score standardization and apply The criterion identifies outliers, uses linear interpolation to replace single point anomalies, and applies moving average method to continuous abnormal data points;

[0018] Set the time window length , construct a state sequence , and calculate statistical characteristics based on this sequence;

[0019] Building a dataset , is a discrete time step, corresponding to each time point in the sampling time series; is the total number of samples, that is, the number of state sequence samples generated by moving the time window during the entire acquisition cycle; add running status labels to the dataset and The set is divided into training set, validation set and test set in the same proportion.

[0020] Furthermore, the operating status tag includes:

[0021] Label 0 indicates normal operation, which satisfies and ;

[0022] Label 1 indicates an attack state that does not trigger protection. The system is under attack but has not exceeded the protection threshold.

[0023] Label 2 indicates the attack state that triggers the frequency protection, which satisfies ;

[0024] Label 3 indicates the attack state that triggers ROCOF protection, which satisfies ;

[0025] in, It represents the frequency change rate, that is, the first-order derivative of the system frequency with respect to time, reflecting the speed of frequency change; Indicates the system frequency deviation, that is, the difference between the current system frequency and the rated frequency; Indicates the maximum allowable frequency deviation. Its value is set according to the stable operating range of the system frequency, and the common value is ±0.5Hz; Indicates the maximum allowable frequency change rate. Its value is set according to the stable operating range of the system frequency, and the common value is ±0.5Hz.

[0026] Furthermore, step 2 is specifically as follows:

[0027] Step 201: Construct a Markov decision process framework for the attacking agent;

[0028] Define the state space as ,in, Indicates the system electrical power, represents mechanical power, represents the regional control error, It is the safe operation area of the system; It is the state representation used by the attacking agent in the reinforcement learning algorithm, which is different from the original running state vector , used for reinforcement learning modeling process;

[0029] Define the action space as , respectively targeting the three key links of sensor, controller and load; among them, Indicates the magnitude of fraudulent manipulation of sensor signals (e.g. frequency measurements), represents the interference intensity on the control input path, Indicates the control parameters of load disturbance characteristics; represents the action space of the attacking agent, which is defined as the set of all possible attack actions, including the control parameters of sensor signals, control paths and load disturbances;

[0030] Define the state transfer equation as ,in is the state transition function, represents the random disturbance term in the system.

[0031] Step 202: Design a multi-objective reward function:

[0032] , the system safe operation area is defined as ;

[0033] in, is the success reward value, is the coefficient, is a parameter; It represents the square of the attack action vector and reflects the magnitude of the attack; is the penalty coefficient in the multi-objective reward function, and its unit is consistent with the reward function. The setting of is a regularization method used to constrain the behavior range of the attacking agent and prevent it from generating physically unreasonable or overly aggressive attack instructions. It helps to achieve a balance between the high concealment and physical feasibility of the attack samples and prevent the attack strategy from falling into unrealistic extreme behaviors.

[0034] Step 203: Use the improved REINFORCE algorithm to optimize the strategy;

[0035] Defining parameterized strategies , indicating that in the state Next select action probability;

[0036] The strategy optimization goal is to maximize the expected cumulative reward:

[0037] ;

[0038] in, represents the attack strategy network parameters, Represents the state-action trajectory, represents the discount factor, Indicates the duration of the attack;

[0039] The policy gradient calculation formula is: ,in, Indicates that from the time step Discounted returns to start; represents the state-based baseline function; Represents the time step index variable, used to calculate the time from the current time step The cumulative value of future rewards at the beginning, its value range is .

[0040] Step 204: Generate a false data injection attack sample and construct an extended state vector containing time period features:

[0041] ,

[0042] in, Represents periodic time feature items, which are used to capture the periodic behavior patterns of the system, enhance the temporal adaptability of the attack, and effectively improve the model's ability to respond to periodic disturbances of the system;

[0043] Attack signal generation uses hyperbolic tangent function to implement physical constraints:

[0044] ,

[0045] The tampered frequency signal received by the controller is:

[0046] ;

[0047] in, It represents the ratio of the tampering amplitude of the frequency signal received by the controller to the real signal. It is the attack action amplitude factor output by the neural network in the current state. Its value range is , control the attack intensity; is the maximum limit of the attack amplitude, which controls the maximum disturbance degree of the tampered signal and ensures that the generated attack signal is physically feasible. This value is a hyperparameter obtained by empirical parameter adjustment and is generally taken as between; represents the attack strategy network; Indicates that the system is at the current time step The true frequency deviation; It represents the frequency measurement value after the attack tampering, that is, the forged signal actually received by the controller, which is multiplied by the proportional coefficient Achieve deception of real signals.

[0048] in, It is a parameterized deep neural network model with an input layer that accepts a real vector of dimension 5; two fully connected hidden layers, each containing 64 neurons, with ReLU activation function; and an output layer with a single neuron and tanh activation function. The attack strategy network input is the expanded state vector , the output is the attack action amplitude factor , whose range is limited to (-1,1) by tanh, and then by Realize disturbance control of frequency measurement signal.

[0049] Step 205: Generate a disturbance attack sample based on load switching, and the attack strategy network outputs three-dimensional gating parameters:

[0050] ,

[0051] in, It represents the central time point of the attack trigger, which is a normalized time index parameter output by the attack strategy network. It actually controls the start timing of the disturbance signal. Its value range is ,in Indicates the maximum attack duration, A larger value indicates a delayed attack trigger, while a smaller value indicates an early trigger. Indicates the steepness of the rise / fall of the disturbance signal, that is, controls the conversion rate of the attack signal. It is the slope control parameter of the Sigmoid function. The larger the value, the steeper the transition of the Sigmoid function and the more sudden the signal change. The smaller the value, the smoother the transition of the attack signal. Its value range is generally a positive real number, for example , which is output by the attack strategy network according to the state adaptively; In order to control the maximum amplitude of the disturbance attack signal, that is, the attack intensity, the adaptive amplitude mapping function Mapped to the physical load disturbance power magnitude; For normalized input, after mapping .

[0052] The attack signal is realized by parameterizing the Sigmoid function and superposition of harmonics:

[0053] ;

[0054] in, The generated disturbance attack signal is a simulation of the load switching event. It acts on the load side of the system and simulates the attacker's behavior of inducing the dynamic response of the system frequency by controlling the load change. is the adaptive amplitude mapping function, which converts the attack strategy network output Mapping is the maximum amplitude of the physical attack signal, in MW (megawatt). This function can use linear or nonlinear mapping forms, for example ,in The upper limit of the maximum disturbance amplitude (e.g. 10MW) is used to control the upper limit of the maximum physical disturbance of the attack signal; For the The amplitude coefficient of each harmonic component, expressed in dimensionless normalization factors or MW, represents the energy contribution of each harmonic in the attack signal. Its value is learned by the attacking agent during training or preset as a tuned value; Indicates the The disturbance signal component of the harmonic order, where For the order disturbance frequency; is the initial phase; this item is used to simulate periodic load disturbance or harmonic interference; this part plays a role in enhancing the resonance response and concealing the attack intention in the entire attack signal;

[0055] The attacking agent can automatically identify the natural frequency of the system and generate disturbances that meet the resonance conditions:

[0056] ,

[0057] in, is the attack disturbance frequency, is the natural frequency of the system, is the allowable error range.

[0058] S206, verifying the validity of the generated sample;

[0059] The attack effect is evaluated by defining the security margin indicator:

[0060] ,

[0061] in, is the minimum Euclidean distance between the current system state and the boundary of the safe area; is the current system state vector, which is taken as the current time step The combination of the system state variables; Indicates a safe area The state vector of any point on the boundary is used to calculate the minimum distance between the current state and the safety boundary; Indicates the entire area the boundaries;

[0062] The attack success criteria are: Or the system state exceeds the safe area at any time during the simulation.

[0063] Furthermore, step 3 is specifically as follows:

[0064] Step 301: Construct a supervised learning detection model based on LSTM to classify and identify attack behaviors in the load frequency control system;

[0065] Step 302: construct an unsupervised learning model based on an autoencoder to detect unknown types of attack behaviors in the load frequency control system;

[0066] Step 303: Based on the detection results of the models in step 301 and step 302, a multi-model fusion detection decision mechanism is designed;

[0067] Step 304: Design a dynamic adjustment mechanism for model parameters to improve the adaptability of the model.

[0068] Furthermore, in step 301, the LSTM-based supervised learning detection model includes an input layer, three LSTM layers, three Dropout layers, a fully connected layer, and a Softmax output layer;

[0069] The input layer receives a sequence of system states constructed within a time window:

[0070] ,

[0071] Among them, the state vector at each time point is:

[0072] ;

[0073] The input sequence is extracted through three stacked LSTM layers for temporal feature extraction. Each LSTM layer is followed by a Dropout layer. The input of each Dropout layer is the output of its corresponding LSTM layer. The outputs of the first two Dropout layers are the input of the next LSTM layer. The output of the third Dropout layer is flattened into a vector form through the Flatten operation and input into the fully connected layer. The output is the intermediate representation of the class probability vector. ; As input, it enters the Softmax output layer to generate the final output probability vector , ; The final predicted category is , the confidence level is ; Represents the supervised learning detection model based on LSTM for the input state sequence The classification prediction result, that is, the input sequence belongs to the category The probability of , corresponding to four types of operating states:

[0074] c=0: normal operating state;

[0075] c=1: attack state that does not trigger protection;

[0076] c=2: triggers the attack state of frequency protection;

[0077] c=3: triggers the attack state of ROCOF protection;

[0078] The sum of the above four types of probabilities satisfies the normalization constraint: .

[0079] Furthermore, in step 302, the unsupervised learning model based on the autoencoder adopts an autoencoder with a symmetrical structure, including an encoder, a potential representation and a decoder;

[0080] The mapping functions of the encoder and decoder are:

[0081] ,

[0082] ;

[0083] in, Represents the network parameter set of the encoder part, which is a model trainable parameter and is used to control the mapping function of the encoder Nonlinear transformation capability; Represents the network parameter set of the decoder part, and Relative, used to control the decoder Reconstruction capabilities; Indicates at time The latent space vector output by the encoder is the input state sequence The present invention adopts a two-dimensional latent space, The encoder compresses the input sequence into a 2-dimensional latent space, and the decoder reconstructs the latent representation into the original sequence.

[0084] The unsupervised learning model training based on the autoencoder adopts the mean square error loss function, and introduces sparse constraints and L2 regularization. The total loss function is:

[0085] ;

[0086] in, Represents the parameter set of the entire autoencoder model (including encoder and decoder), that is , used to uniformly represent model complexity in the regularization term; Represents the mean square error loss function between the input sequence and the reconstructed sequence, which measures the reconstruction error of the model; represents the weight hyperparameter of the sparse regularization term, which is used to control the latent representation Sparsity to prevent model overfitting; express The weight hyperparameter of the regularization term is used to control the model parameters The norm size of the model is used to avoid instability in training caused by excessively large model parameters;

[0087] The formula for calculating the anomaly detection threshold is: ;

[0088] in, and are the mean and standard deviation of the reconstruction error, respectively; Represents a hyperparameter used to adjust the sensitivity of anomaly detection, with a value range of , which can be adjusted according to the actual system's tolerance for false alarm rate / missing alarm rate;

[0089] The formula for calculating the anomaly score is: ;

[0090] in, is the reconstruction error, is the anomaly detection threshold; for a new input sequence, if the reconstruction error , it is judged as abnormal.

[0091] Furthermore, step 303 is specifically as follows:

[0092] Step 303-1: Obtain the detection results of the LSTM-based supervised learning detection model and the autoencoder-based unsupervised learning model, including the output category probability vector of the supervised learning detection model. , supervised learning detection model determines the category , supervised learning detection model determines confidence , unsupervised model anomaly score And the confidence of unsupervised model determination ;

[0093] Step 303-2: Dynamically assign decision weights based on historical performance of each model:

[0094] ,

[0095] ,

[0096] in, and Respectively indicate time The weights of the LSTM-based supervised learning detection model and the autoencoder-based unsupervised learning model; and Represents the supervised learning detection model based on LSTM and the unsupervised learning model based on autoencoder at time The accuracy within the previous sliding window;

[0097] Calculate the weighted score of the fusion decision:

[0098] ;

[0099] in, and are the confidence scores of the two models output corresponding to time t;

[0100] Step 303-3: Design fusion decision rules:

[0101] When the two models are consistent, the consistent result is directly adopted;

[0102] When the judgment is inconsistent, if and ,using supervised learning to detect model results;

[0103] in, Represents the decision threshold for the confidence of the supervised learning detection model; Represents the judgment threshold of the confidence of the unsupervised learning model; the two are adjustable hyperparameters set by the system, mainly used to control the boundary of the confidence of different models in the fusion decision rule; and The typical value range of , the tolerance for false positives / missing negatives can be empirically adjusted based on the actual system; during the deployment phase, parameters can be adjusted through the validation set or heuristic strategy settings, and dynamic optimization can also be performed through model performance feedback during operation;

[0104] like and , using unsupervised learning model results;

[0105] Otherwise, according to With preset threshold Compare and determine the final judgment.

[0106] Weighted scores for multiple models The fusion decision threshold is used to independently guide the final decision when the confidence does not significantly favor any party. As the only independent confidence boundary in the fusion path, its setting needs to be tuned based on the statistical performance indicators on the model training set or validation set. In practical applications, this can be achieved by analyzing the Receiver Operating Characteristic (ROC) curve and selecting the point corresponding to the optimal F1 score. value, or by constructing a joint loss function of false positives and false negatives, The optimal value of is obtained, thereby suppressing the false positive rate while maintaining the detection sensitivity.

[0107] when and , that is, both models have not reached their set confidence thresholds, and the system cannot make independent decisions based on the judgment of any single model; in this case, the system switches to a fusion scoring mechanism, by constructing a weighted confidence score Comprehensive evaluation of multiple model outputs.

[0108] like , the system believes that the confidence level after multi-model integration has reached the acceptable threshold, and the output state is determined to be "attack state"; further, the system The relative proportion of the confidence contribution of each sub-model in determines the attack type. For example, when When the weighted score accounts for the largest proportion and the current prediction label of the model is 3, the system will finally output "attack status triggering frequency protection" as the judgment result; on the contrary, when When , the overall confidence level is insufficient to support abnormal judgment, and the system outputs "normal operation status".

[0109] Furthermore, step 304 is specifically as follows:

[0110] Implement the learning rate adaptive adjustment strategy:

[0111] ,

[0112] in, express The learning rate at each moment, represents the initial learning rate, represents the attenuation coefficient; Indicates the maximum number of training rounds; defines the validation set performance indicators , set the patience parameter , if continuous After round training If there is no improvement, stop training and return the model parameters with the best performance on the validation set;

[0113] Build a dynamic update strategy for model weights and define the update frequency , set the update limit To prevent model mutations, adjust the weight update direction based on the performance feedback of new data batches and implement a soft update mechanism: ,in represents the soft update coefficient, ;

[0114] Develop an adaptive detection threshold adjustment algorithm to dynamically adjust the detection threshold based on historical false alarm rates and missed alarm rates, and design a threshold adjustment formula: ,

[0115] in, and represent the new and old detection thresholds respectively; Indicates the adjustment step size; Indicates the current false alarm rate; Indicates the target false alarm rate.

[0116] In existing anomaly detection methods, the commonly used detection threshold setting methods are mostly fixed thresholds, statistical distribution estimation (such as based on standard deviation), The threshold adaptive adjustment algorithm proposed in this paper introduces the false positive rate (FPR) as a feedback variable into the threshold adjustment formula for the first time. By applying a multiplicative adjustment to the difference between the target FPR and the current FPR, the algorithm achieves dynamic adaptive optimization of detection sensitivity, drawing on the negative feedback mechanism in control theory.

[0117] Furthermore, step 4 is specifically as follows:

[0118] Step 401: Perform attack detection based on the test sample;

[0119] Load test sample set ,in Indicates the The feature vector of the test sample, For the corresponding label, Indicates the number of the test sample, is the total number of test set samples;

[0120] Calculate the class probability vector through supervised learning detection model ,in represents the supervised learning model detection function, are model parameters, Indicates the test samples belong to the category The predicted probability value of Corresponding to four types of operating states;

[0121] Determine the predicted category of the supervised learning detection model based on the probability vector and confidence ; At the same time, the reconstruction error is calculated through the unsupervised learning model ,in represents the autoencoder model function, is the corresponding parameter;

[0122] Based on the mean of the reconstruction error , standard deviation , anomaly detection threshold , calculate the normalized anomaly score: ;

[0123] Determine the abnormality result based on the reconstruction error score And confidence: ;

[0124] Step 402: Calculate the fusion decision score based on the historical performance of the model and determine the final decision result, specifically including:

[0125] Step 402-1: In the model evaluation sliding window The historical classification accuracy of the supervised learning model and the unsupervised model are calculated separately: and , indicating the deadline Model accuracy within the local window of ;

[0126] Calculate the fusion weight of the two detection models at the current time point based on the accuracy:

[0127] ;

[0128] Step 402-2: test samples, constructing the fusion decision score :

[0129] ,

[0130] in, Indicates an indicator function, which returns 1 if the condition in the brackets is true, otherwise it returns 0; Represents the supervised model for the sample If the predicted category , indicating that the model does not output empty categories; is the judgment result of the unsupervised model. When , it means that the sample is detected to be abnormal; the variable and Respectively represent the confidence of the two models in their output results and the weight coefficient and The two models are dynamically adjusted based on their performance within the historical time window to reflect their relative credibility;

[0131] In determining the final fusion classification results If the judgments of the two models are consistent, that is, , then the consistent result is directly used as the final output. If the two results are inconsistent, the priority determination is performed based on their respective confidence levels and preset thresholds. The specific situation is as follows: and When , the prediction result of the supervised model is used as the final classification, that is, On the contrary, if and , then the judgment result of the unsupervised model is adopted, that is, ;

[0132] When the confidence of the two models is not enough to reach their respective set thresholds, that is, and , then by comparing the fusion scores and decision threshold To complete the final judgment. , then it is considered that the sample has attack behavior, that is, Otherwise, it is judged to be in normal state, i.e. ;in, and are the minimum confidence thresholds of the supervised model and the unsupervised model, respectively, used to screen credible outputs; It is the global threshold used for decision making in low confidence situations during the fusion decision process;

[0133] S403. Evaluate and optimize model performance, calculate confusion matrix and false alarm rate and false negative rate , draw the ROC curve and determine the optimal detection threshold, and implement the detection threshold adaptive adjustment algorithm ;

[0134] Step 404: Implement incremental learning and model updating;

[0135] Construct a high-confidence sample collection mechanism, and trigger the update of supervised learning detection model parameters when the number of new samples reaches the preset threshold; implement the online update mechanism of the unsupervised learning model, and use the exponential moving average method to update the reconstruction error statistical parameters. The mean update formula is: , the standard deviation update formula is ,in is the smoothing factor; realize the soft update mechanism of model parameters ,in is the soft update coefficient;

[0136] Step 405: Perform system safety margin assessment and deployment integration, and calculate system safety margin indicators. ,in is the maximum allowable frequency deviation, The maximum allowable frequency change rate is used; based on the safety margin, the system risk level is evaluated and defense strategy recommendations are generated to provide decision support for system operators.

[0137] The beneficial effects of the present invention are:

[0138] 1) This invention uses a reinforcement learning-based attack agent to automatically generate highly concealed and complex network attack samples in LFC systems by constructing a Markov decision process model and designing a multi-objective reward function. The attack agent comprehensively considers system dynamics such as frequency deviation, frequency change rate, and regional control error, and generates multidimensional attack signals targeting sensors, controllers, and load components. This significantly improves the diversity, realism, and threat level of adversarial samples, addressing the lack of systematic attack sample generation and difficulty covering complex attack scenarios in existing methods. This provides more comprehensive and realistic attack scenario support for training and evaluating detection models.

[0139] 2) This invention is based on a dual-modal detection architecture that combines supervised learning with unsupervised learning. By integrating the LSTM deep learning model with the autoencoder model, it achieves accurate identification of known attacks and effective detection of unknown attacks. The LSTM model can extract deep features from time series and capture the dynamic evolution of attack behavior; the autoencoder model establishes a reconstructed error distribution based on normal state data to identify abnormal patterns in unseen attack samples. Through an adaptive weight fusion mechanism, decision weights are dynamically allocated based on the historical performance of the model, improving the accuracy, robustness, and generalization ability of the detection system, effectively solving the problem of weak unknown attack recognition and insufficient generalization capabilities of traditional methods.

[0140] 3) This invention integrates an incremental learning mechanism with a dynamic model parameter adjustment strategy. Through methods such as high-confidence sample selection, model soft updates, and adaptive threshold adjustment, it achieves continuous learning and performance optimization during the detection system's operation. The system can dynamically update the model structure and parameters based on new attack samples and changes in the operating environment, avoiding the detection performance degradation caused by static model deployment and enhancing the model's ability to cope with attack evolution and system state changes. This enables long-term stable and highly adaptable attack detection, overcoming the short lifecycle and lack of update mechanisms of existing methods.

[0141] This invention generates high-quality attack samples through reinforcement learning, improves recognition capabilities through a dual-modal detection architecture, and enhances system adaptability through a continuous learning mechanism. It comprehensively improves the detection accuracy, system robustness, and operational stability of the LFC system in the face of complex network attacks, and significantly enhances the security protection level of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0142] Figure 1 A flow chart of the method of the present invention;

[0143] Figure 2 This is a schematic diagram of the microgrid test platform structure diagram;

[0144] Figure 3It is a line graph for analyzing the frequency dynamic response characteristics under load disturbance;

[0145] Figure 4 Comparison chart of reward functions for power system attacks;

[0146] Figure 5 This is the microgrid attack response and system stability analysis diagram;

[0147] Figure 6 Comparison of tampering attack characteristics and system responses for different frequency measurements of microgrid MG2;

[0148] Figure 7 Performance analysis diagram of autoencoder signal reconstruction;

[0149] Figure 8 Comparison of the accuracy of the multi-model fusion detection mechanism under different attack modes. DETAILED DESCRIPTION

[0150] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0151] like Figure 1 As shown, the load frequency control system attack detection method based on reinforcement learning described in the present invention includes the following steps:

[0152] Step 1: Based on the key state variables and parameters of the load frequency control system, with the goal of building a complete data set, a standardized training set, validation set, and test set are constructed through multi-stage data collection and processing;

[0153] Step 2: Based on the Markov decision process theory framework, with the goal of generating efficient and covert attack samples, we train the attack agent by constructing a multi-objective reward function and an improved REINFORCE algorithm to construct a set of false data injection and load switching attack samples;

[0154] Step 3: Based on time series features and system status data, with the goal of achieving accurate and reliable attack detection, an attack detection model is constructed that includes a supervised learning detection model, an unsupervised learning model, and a fusion decision mechanism.

[0155] Step 4: Based on the trained attack detection model, build an attack detection and defense system with the goal of optimizing detection performance and system adaptability.

[0156] The method described in the present invention takes reinforcement learning as its core and combines multiple machine learning technologies to systematically implement the complete process from data construction, attack simulation, detection modeling to defense implementation, reflecting intelligent and automated attack detection and defense capabilities.

[0157] This embodiment provides a specific implementation of a reinforcement learning-based load frequency control system attack detection method and comprehensively verifies the effectiveness of the proposed method in a microgrid simulation environment. The purpose of the simulation is to evaluate the impact of reinforcement learning-driven attack strategies on the frequency stability of the power system and the performance of the multi-model fusion detection mechanism in identifying complex attack behaviors. To achieve the above goals, the simulation experiment was completed in the MATLAB / Simulink environment, and the Simscape toolbox was used to model the main components of the microgrid system. The test platform was designed and constructed based on a 2.5MVA microgrid with a linear topology consisting of 8 busbar nodes (B1 to B8). The system includes synchronous generators, load nodes, circuit breakers, and frequency protection devices. The LFC parameters of multiple regions are configured to form a diverse test system (MG1 to MG3).

[0158] This example simulates the dynamic changes in system load using a Gaussian random walk model, constructs frequency response data under real operating conditions, and designs two typical attack scenarios: an FDI attack and a load switching attack. The attack strategy is trained using a deep reinforcement learning algorithm based on the potential distance heuristic. Its goal is to induce the tripping of system protection devices by minimizing system frequency deviation and maximizing the frequency change rate, thereby undermining system stability. For attack detection, a supervised learning model based on LSTM and an unsupervised learning model based on autoencoders were constructed, and further integrated to form a multi-model fusion detection mechanism. These models were trained and tested on a comprehensive dataset encompassing multiple operating states to evaluate their performance across multiple performance metrics, including accuracy, false alarm rate, and detection latency. Experimental results demonstrate that the proposed method outperforms existing detection mechanisms in both detection accuracy and attack defense effectiveness, demonstrating its high engineering application value.

[0159] like Figure 2 Figure 2 shows the structure of the microgrid test platform constructed in this embodiment. This platform simulates a typical low-inertia distributed power system. The main power source is connected to the left side of the system, with busbar nodes connected in series to form a backbone network. The key power generation equipment, the synchronous generator, is connected to bus B6, the primary target of FDI attacks. Other busbar nodes are loaded with different loads and serve as targets for load-switching attacks. Circuit breakers (CB1 and CB2) are configured in the system to simulate the operational response of protective devices. This structure provides a realistic and controllable operating environment for subsequent attack experiments and detection mechanism evaluation.

[0160] In this embodiment, the performance of three test microgrid systems MG1, MG2 and MG3 is analyzed and compared from the perspective of six key control parameters (see Table 1).

[0161] Table 1 Comparison of main parameters of three microgrid systems

[0162]

[0163] MG3 has the highest values for Automatic Generation Control (AGC) gain and generator inertia, at 12 and 8, respectively, demonstrating significant advantages in frequency control and disturbance rejection. In contrast, MG1 has the lowest values for these two parameters, with an AGC gain of only 2 and a generator inertia of 3, indicating weaker stability and regulation capabilities. MG2's values fall between those of MG1 and MG3, with an AGC gain of 8 and an inertia of 6. In terms of droop gain, both MG2 and MG3 have a value of 50, surpassing MG1's 40, indicating smoother frequency response. In terms of governor time constant, MG1 has the lowest value of 0.6, while MG2 and MG3 have values of 0.7 and 0.8, respectively, indicating that MG1 has the fastest response. Turbine time constants vary slightly, with MG1 having a slightly lower value of 0.9 and both MG2 and MG3 having values of 1.0. In terms of damping constant, MG3 has the highest value of 0.9, MG2 has a value of 0.7, and MG1 has the lowest value of 0.5, indicating that MG3 has stronger oscillation suppression capabilities. Overall, MG3 performed best across all metrics, MG2 had balanced performance, and MG1 was relatively weak. These parameter differences laid the foundation for subsequent research to analyze the transferability of attack strategies and the universality of detection mechanisms.

[0164] Figure 3 This line chart analyzes the dynamic response characteristics of the frequency under load disturbances, showing the dynamic relationship between the load power change (blue solid line), the frequency deviation (orange-red solid line), and its rate of change (purple dashed line) over time. It is clearly observed that at t = 300s, 600s, and 1000s, the load power undergoes dramatic changes, and the system frequency deviation also undergoes sudden changes simultaneously. The frequency change rate increases rapidly, and the frequency response exhibits clear dynamic disturbance characteristics. These mutation points are represented by red dots, further highlighting the system's sensitive response to load disturbances. The maximum correlation coefficient between the load power change and the frequency deviation, shown in the lower right corner of the figure, reaches 0.92, and the optimal lag time is 0 seconds, indicating a high degree of synchronization and strong correlation between the two.

[0165] Figure 4 This is a comparison chart of the reward function for power system attacks, showing the reward function distribution of two intelligent attack strategies: FDI attack and load switching attack, and presenting the reward value distribution characteristics of different attack strategies on the frequency-frequency change rate plane. Figure 4In the figure, circular bubbles represent the reward distribution of the FDI attack, while diamond bubbles represent the reward distribution of the load-switching attack. The size and color of the bubbles indicate the magnitude of the reward, and the red border indicates the frequency protection threshold. The reward distribution of the FDI attack (circular bubbles) shows that the reward is primarily concentrated in regions with large frequency variation rates, with higher rewards near the frequency protection threshold (red border). This indicates that the FDI attack strategy tends to produce large frequency variation rates with small frequency deviations, thereby achieving a highly concealed and destructive attack effect. The load-switching attack (diamond bubbles) exhibits a distinct elliptical symmetric distribution, with the reward gradually increasing from the center outward, forming a "risk ring" structure. This design encourages the attacking agent to push the system state toward the frequency protection threshold region, thereby triggering system protection actions. The reward function designs of the two attacks reflect different attack objectives and strategies. The FDI attack focuses more on concealment and specific system state deviations, while the load-switching attack emphasizes a holistic strategy of pushing the system toward the instability boundary.

[0166] Figure 5 The following figure shows a comparison of the dynamic response characteristics of microgrid systems with different eigenmode frequencies (MG2: 4 rad / s and MG3: 3.4 rad / s) when subjected to a reinforcement learning attack strategy. The figure uses a combination of area and bubble plots to comprehensively visualize the correlation between the system stability evolution and key parameters. The bubble size represents the absolute value of the ROCOF, and the curves show the cumulative stability impact and attack energy. Experimental data shows that the attack signal begins to act on the system at t=1s, after which both microgrid systems exhibit significant frequency shifts and stability degradation. The cumulative stability impact index of the MG2 system (blue area) increases significantly faster than that of the MG3 system (red area), demonstrating the important characteristic that systems with high eigenfrequency frequencies are more sensitive to interference. The distribution of bubbles at the frequency-ROCOF characteristic points further confirms this conclusion. The bubble size (representing the absolute value of the ROCOF) of the MG2 system is generally larger than that of the MG3 system, indicating a more significant rate of frequency change. Analysis of the triggering time of the protection devices shows that the ROCOF protection of the MG2 system was triggered at t≈1.73s, while the MG3 system was delayed until t≈2.15s, a time difference of approximately 0.42s. This result is consistent with the theoretical rapid response characteristics of high eigenmode frequency systems to disturbances. The attack energy curve (dashed line) shows that although the energy accumulation trends of the attack signals received by the two systems are similar, their system responses differ significantly, confirming that the reinforcement learning attack strategy can generate targeted interference patterns based on system characteristics.

[0167] Figure 6The following figure compares the characteristics of different frequency measurement tampering attacks on microgrid MG2 and the system response. The impact characteristics of three types of frequency measurement tampering attacks (low-frequency, medium-frequency, and high-frequency) on microgrid MG2 are compared. The results show that although the three attacks have the same amplitude (0.1 unit), the medium-frequency attack (0.8Hz, lasting 6 seconds) causes the largest frequency deviation (2.0Hz) and the largest frequency change rate (3.2Hz / s), exceeding the system frequency change rate protection threshold (3Hz / s). The high-frequency attack (1.5Hz, lasting 15 seconds), while having a smaller frequency deviation (0.8Hz), has the highest number of threshold violations (7 times), indicating cumulative damage. The low-frequency attack (0.5Hz, lasting 4 seconds) has a milder impact, with all indicators within the limit. Experiments confirm that the supervised detection model proposed in this study achieves 99.2% detection accuracy for medium-frequency attacks and can complete identification and trigger defense within 0.15 seconds, effectively controlling system parameters within safety thresholds. This demonstrates the method's high real-time and defense efficiency. For measurement tampering attacks with different frequency characteristics, we should pay attention to the instantaneous impact of medium-frequency attacks and the cumulative effects of high-frequency attacks, and improve the security protection capabilities of microgrids through differentiated defense strategies.

[0168] Figure 7 This is a performance analysis diagram of the autoencoder signal reconstruction, showing the autoencoder's response to frequency deviation ( ) and control error ( ) signal reconstruction effect comparison. In the figure, the blue solid line and dotted line represent the true value and reconstructed value of the frequency deviation, respectively, and the red solid line and dotted line represent the true value and reconstructed value of the control error, respectively. The system fluctuates at multiple moments (0.8s, 2.2s, 3.8s, 5.2s, 8.2s and 10.0s), among which the disturbance at 5.2s is the most significant. The reconstruction results show that the autoencoder can accurately capture the dynamic characteristics of the two types of signals. The standard deviation of the frequency deviation signal reconstruction error is Per unit value, the standard deviation of the reconstruction error of the control error signal is Even when the system fluctuates violently, the reconstruction error remains at a low level, verifying the effectiveness of the model in power system signal reconstruction.

[0169] Figure 8The chart comparing the accuracy of the multi-model fusion detection mechanism under different attack modes shows the detection accuracy of the multi-model fusion detection mechanism compared with that of a single supervised model and a single unsupervised model under three different load switching attack modes: low-frequency attack (0.5Hz, 4 seconds), medium-frequency attack (0.8Hz, 6 seconds), and high-frequency attack (1.5Hz, 15 seconds). The results show that the fusion mechanism achieved detection accuracies of 98.5%, 99.1%, and 99.9% in these three modes, respectively, outperforming the single supervised model (94.2%, 95.8%, and 97.3%) and the single unsupervised model (92.6%, 93.5%, and 95.2%). For the square wave attack mode C, the fusion mechanism achieved a detection accuracy of 99.9%. The fusion mechanism significantly reduced the system frequency deviation and frequency change rate without exceeding the safety threshold, verifying the effectiveness of the mechanism across various attack modes.

[0170] This example systematically validates the effectiveness of the proposed method for attack detection and defense in load frequency control systems by constructing a multi-parameter microgrid simulation platform and combining attack strategies generated by a reinforcement learning agent with a multi-model fusion detection mechanism. Experimental results demonstrate that this method outperforms existing technologies in detection accuracy, false alarm control, attack adaptability, and real-time response, demonstrating its engineering feasibility and potential for widespread application.

[0171] The above description is only a preferred embodiment of the present invention and is not intended to further limit the present invention. All equivalent changes made using the contents of the present invention description and drawings are within the scope of protection of the present invention.

Claims

1. A load frequency control system attack detection method based on reinforcement learning, characterized in that: The following steps are involved: Step 1: Based on the key state variables and parameters of the load frequency control system, with the goal of building a complete data set, a standardized training set, validation set, and test set are constructed through multi-stage data collection and processing; Step 2: Based on the Markov decision process theory framework, with the goal of generating efficient and covert attack samples, we train the attack agent by constructing a multi-objective reward function and an improved REINFORCE algorithm to construct a set of false data injection and load switching attack samples; Step 3: Based on time series features and system status data, with the goal of achieving accurate and reliable attack detection, an attack detection model is constructed that includes a supervised learning detection model, an unsupervised learning model, and a fusion decision mechanism. Step 4: Based on the trained attack detection model, build an attack detection and defense system with the goal of optimizing detection performance and system adaptability; Among them, step 2 is specifically as follows: Step 201: Construct a Markov decision process framework for the attacking agent; Define the state space as ,in, Indicates the system electrical power, represents mechanical power, represents the regional control error, This is the safe operating area of the system. is the system frequency deviation, is the transpose of the vector, Indicates the rate of change of frequency; Define the action space as ,in, represents the magnitude of the fraudulent tampering of the sensor signal, represents the interference intensity on the control input path, Indicates the control parameters of load disturbance characteristics; represents the action space of the attacking agent, which is defined as the set of all possible attack actions, including the control parameters of sensor signals, control paths and load disturbances; Define the state transfer equation as ,in is the state transition function, represents the random disturbance term in the system, It is the system control input; Step 202: Design a multi-objective reward function: ， The system safe operation area is defined as ; in, is the success reward value, is the coefficient, is a parameter; is the penalty coefficient in the multi-objective reward function; It represents the square of the attack action vector and reflects the magnitude of the attack; Step 203: Use the improved REINFORCE algorithm to optimize the strategy; Defining parameterized strategies , indicating that in the state Next select action probability; The strategy optimization goal is to maximize the expected cumulative reward: ； in, represents the policy network parameters, Represents the state-action trajectory, represents the discount factor, Indicates the duration of the attack. is a multi-objective reward function; The policy gradient calculation formula is: ， in, Indicates that from the time step Discounted returns to start; represents the state-based baseline function; Represents the time step index variable, used to calculate the time from the current time step The cumulative value of future rewards at the beginning, its value range is ; Step 204: Generate a false data injection attack sample and construct an extended state vector containing time period features: ， in, Represents the periodic time feature item, which is used to capture the periodic behavior pattern of the system and enhance the temporal adaptability of the attack; Attack signal generation uses hyperbolic tangent function to implement physical constraints: ， The tampered frequency signal received by the controller is: ； in, Indicates the ratio of the tampered amplitude of the frequency signal received by the controller to the real signal; Represents the attack strategy network, whose input is the expanded state vector , the output is the scale factor inside the attack ratio function, Its network parameters; Indicates that the system is at the current time step The true frequency deviation; It represents the frequency measurement value after the attack tampering, that is, the forged signal actually received by the controller, which is multiplied by the proportional coefficient Achieve deception of real signals; The maximum limit of the attack range; Step 205: Generate a disturbance attack sample based on load switching, and the attack strategy network outputs three-dimensional gating parameters: ， in, Indicates the central time point when the attack is triggered; Indicates the steepness of the rise / fall of the disturbance signal, that is, controls the conversion rate of the attack signal; To control the maximum amplitude of the disturbance attack signal, that is, the attack intensity; The attack signal is realized by parameterizing the Sigmoid function and superposition of harmonics: ； in, is the generated disturbance attack signal, is the adaptive amplitude mapping function, which converts the attack strategy network output Mapped to the maximum amplitude of the physical attack signal; For the Amplitude coefficients of harmonic components; Indicates the The disturbance signal component of the harmonic order, where For the order disturbance frequency; is the initial phase; The attacking agent can automatically identify the natural frequency of the system and generate disturbances that meet the resonance conditions: ， in, is the attack disturbance frequency, is the natural frequency of the system, is the allowable error range; S206, verifying the validity of the generated sample; The attack effect is evaluated by defining the security margin indicator: ， in, is the minimum Euclidean distance between the current system state and the boundary of the safe area; is the current system state vector, which is taken as the current time step The combination of the system state variables; Indicates a safe area The state vector of any point on the boundary is used to calculate the minimum distance between the current state and the safety boundary; Indicates the entire area the boundaries; The attack success criteria are: Or the system state exceeds the safe area at any time during the simulation.

2. The load frequency control system attack detection method based on reinforcement learning according to claim 1 is characterized in that: Step 1 is as follows: The key state variables of the load frequency control system are collected at a preset sampling frequency to obtain the system state vector ,in, is the frequency integration error, is the speed regulator power deviation, is the mechanical power deviation, is the system frequency deviation, is the transpose of the vector; at the same time, the control input is collected and load disturbances , and key system parameters , including the governor time constant , turbine time constant , system inertia constant , damping coefficient and the governor droop coefficient ; For all sampled dynamic variables, including system state variables 、 、 、 , and system control input and load disturbances , Z-score standardization method was applied for processing; Clean the data after Z-score standardization and apply The criterion identifies outliers, uses linear interpolation to replace single point anomalies, and applies moving average method to continuous abnormal data points; Set the time window length , construct a state sequence , and calculate statistical characteristics based on this sequence; Building a dataset , It is the key parameter of the system; is a discrete time step corresponding to each time point in the sampling time series; N is the total number of samples, that is, the number of state sequence samples generated by moving the time window during the entire acquisition cycle; add running status labels to the data set and follow The set is divided into training set, validation set and test set in the same proportion.

3. The load frequency control system attack detection method based on reinforcement learning according to claim 2 is characterized in that: The running status tag includes: Label 0 indicates normal operation, which satisfies and ; Label 1 indicates an attack state that does not trigger protection. The system is under attack but has not exceeded the protection threshold. Label 2 indicates the attack state that triggers the frequency protection, which satisfies ; Label 3 indicates the attack state that triggers ROCOF protection, which satisfies ; in, It represents the frequency change rate, that is, the first-order derivative of the system frequency with respect to time, reflecting the speed of frequency change; Indicates the system frequency deviation, that is, the difference between the current system frequency and the rated frequency; Indicates the maximum allowable frequency deviation; Indicates the maximum allowable frequency change rate.

4. The load frequency control system attack detection method based on reinforcement learning according to claim 1 is characterized in that: Step 3 is as follows: Step 301: Construct a supervised learning detection model based on LSTM to classify and identify attack behaviors in the load frequency control system; Step 302: construct an unsupervised learning model based on an autoencoder to detect unknown types of attack behaviors in the load frequency control system; Step 303: Based on the detection results of the models in step 301 and step 302, a multi-model fusion detection decision mechanism is designed; Step 304: Design a dynamic adjustment mechanism for model parameters to improve the adaptability of the model.

5. The load frequency control system attack detection method based on reinforcement learning according to claim 4 is characterized in that: In step 301, the LSTM-based supervised learning detection model includes an input layer, three LSTM layers, three Dropout layers, a fully connected layer, and a Softmax output layer; The input layer receives a sequence of system states constructed within a time window: ， Among them, the state vector at each time point is: ； The input sequence is extracted through three stacked LSTM layers for temporal feature extraction. Each LSTM layer is followed by a Dropout layer. The input of each Dropout layer is the output of its corresponding LSTM layer. The outputs of the first two Dropout layers are the input of the next LSTM layer. The output of the third Dropout layer is flattened into a vector form through the Flatten operation and input into the fully connected layer. The output is the intermediate representation of the class probability vector. ; As input, it enters the Softmax output layer to generate the final output probability vector ,in ; The final predicted category is , the confidence level is ; LSTM-based supervised learning detection model for input state sequence The classification prediction result, that is, the input sequence belongs to the category The probability of ; corresponding to four types of operating states: c=0: normal operating state; c=1: attack state that does not trigger protection; c=2: triggers the attack state of frequency protection; c=3: Triggers the attack state of ROCOF protection.

6. The load frequency control system attack detection method based on reinforcement learning according to claim 5 is characterized in that: In step 302, the unsupervised learning model based on the autoencoder adopts an autoencoder with a symmetrical structure, including an encoder, a potential representation and a decoder; The mapping functions of the encoder and decoder are: ， ； in, Represents a set of network parameters for the encoder part, used to control the encoder's mapping function Nonlinear transformation capability; Represents a set of network parameters for the decoder part, used to control the decoder Reconstruction capabilities; Indicates at time The latent space vector output by the encoder is the input state sequence Low-dimensional embedding representation of ; The unsupervised learning model training based on the autoencoder adopts the mean square error loss function, and introduces sparse constraints and L2 regularization. The total loss function is: ； in, Represents the parameter set of the entire autoencoder model, that is ; Represents the mean square error loss function between the input sequence and the reconstructed sequence, which measures the reconstruction error of the model; represents the weight hyperparameter of the sparse regularization term, which is used to control the latent representation Sparsity to prevent model overfitting; express The weight hyperparameter of the regularization term is used to control the model parameters The norm size of the model is used to avoid instability in training caused by excessively large model parameters; The formula for calculating the anomaly detection threshold is: ; in, and are the mean and standard deviation of the reconstruction error, respectively; represents the hyperparameter used to adjust the sensitivity of anomaly detection; The formula for calculating the anomaly score is: ; in, is the reconstruction error, is the anomaly detection threshold; for a new input sequence, if the reconstruction error , it is judged as abnormal.

7. The load frequency control system attack detection method based on reinforcement learning according to claim 6 is characterized in that: Step 303 is specifically as follows: Step 303-1: Obtain the detection results of the LSTM-based supervised learning detection model and the autoencoder-based unsupervised learning model, including the output category probability vector of the supervised learning detection model. , supervised learning detection model determines the category , supervised learning detection model determines confidence , unsupervised model anomaly score And the confidence of unsupervised model determination ; Step 303-2: Dynamically assign decision weights based on historical performance of each model: ， ， in, and Respectively indicate time The weights of the LSTM-based supervised learning detection model and the autoencoder-based unsupervised learning model; and Represents the supervised learning detection model based on LSTM and the unsupervised learning model based on autoencoder at time The accuracy within the previous sliding window; Calculate the weighted score of the fusion decision: ； Step 303-3: Design fusion decision rules: When the two models are consistent, the consistent result is directly adopted; When the judgment is inconsistent, if and ,using supervised learning to detect model results; in, Represents the decision threshold for the confidence of the supervised learning detection model; Represents the decision threshold for the confidence of the unsupervised learning model; like and , using unsupervised learning model results; Otherwise, according to With preset threshold Compare and determine the final judgment; like , the system believes that the confidence level after multi-model integration has reached the acceptable threshold, and the output state is determined to be an attack state; when , then the overall confidence level is not enough to support abnormal judgment, and the system outputs normal operating status.

8. The load frequency control system attack detection method based on reinforcement learning according to claim 7 is characterized in that: Step 304 is specifically as follows: Implement the learning rate adaptive adjustment strategy: ， in, express The learning rate at each moment, represents the initial learning rate, represents the attenuation coefficient; Indicates the maximum number of training rounds; defines the validation set performance indicators , set the patience parameter , if continuous After round training If there is no improvement, stop training and return the model parameters with the best performance on the validation set; Build a dynamic update strategy for model weights and define the update frequency , set the update limit To prevent model mutations, adjust the weight update direction based on the performance feedback of new data batches and implement a soft update mechanism: ,in represents the soft update coefficient, ; Develop an adaptive detection threshold adjustment algorithm, dynamically adjust the detection threshold based on historical false alarm rates and missed alarm rates, and design a threshold adjustment formula , in, and represent the new and old detection thresholds respectively; Indicates the adjustment step size; Indicates the current false alarm rate; Indicates the target false alarm rate.

9. The load frequency control system attack detection method based on reinforcement learning according to claim 8 is characterized in that: Step 4 is as follows: Step 401: Perform attack detection based on the test sample; Load test sample set ,in Indicates the The feature vector of the test sample, For the corresponding label, Indicates the number of the test sample, is the total number of test set samples; Calculate the class probability vector through supervised learning detection model ,in represents the supervised learning model detection function, are model parameters, Indicates the test samples belong to the category The predicted probability value of Corresponding to four types of operating states; Determine the predicted category of the supervised learning detection model based on the probability vector and confidence ; At the same time, the reconstruction error is calculated through the unsupervised learning model ,in represents the autoencoder model function, is the corresponding parameter; Based on the mean of the reconstruction error , standard deviation , anomaly detection threshold , calculate the normalized anomaly score: ; Determine the abnormality result based on the reconstruction error score And confidence: ； Step 402: Calculate the fusion decision score based on the historical performance of the model and determine the final decision result, specifically including: Step 402-1: In the model evaluation sliding window The historical classification accuracy of the supervised learning model and the unsupervised model are calculated separately: and , represents the model accuracy in the local window up to time t; Calculate the fusion weight of the two detection models at the current time point based on the accuracy: ， Step 402-2: test samples, constructing the fusion decision score : ， in, Indicates an indicator function, which returns 1 if the condition in the brackets is true, otherwise it returns 0; Represents the supervised model for the sample If the predicted category , indicating that the model does not output empty categories; is the judgment result of the unsupervised model. When , it means that the sample is detected to be abnormal; the variable and Respectively represent the confidence of the two models in their output results and the weight coefficient and The two models are dynamically adjusted based on their performance within the historical time window to reflect their relative credibility; In determining the final fusion classification results If the judgments of the two models are consistent, that is, , then the consistent result is directly used as the final output; if the two results are inconsistent, the priority determination is performed based on their respective confidence levels and preset thresholds; When the confidence of the two models is not enough to reach their respective set thresholds, the fusion score is compared. and decision threshold To complete the final judgment; S403. Evaluate and optimize model performance, calculate confusion matrix and false alarm rate and false negative rate , draw the ROC curve and determine the optimal detection threshold, and implement the detection threshold adaptive adjustment algorithm ; Step 404: Implement incremental learning and model updating; Construct a high-confidence sample collection mechanism, and trigger the update of supervised learning detection model parameters when the number of new samples reaches the preset threshold; implement the online update mechanism of the unsupervised learning model, and use the exponential moving average method to update the reconstruction error statistical parameters. The mean update formula is: , the standard deviation update formula is ,in is the smoothing factor; realize the soft update mechanism of model parameters ,in is the soft update coefficient; Step 405: Perform system safety margin assessment and deployment integration, and calculate system safety margin indicators. ,in is the maximum allowable frequency deviation, The maximum allowable frequency change rate is used; based on the safety margin, the system risk level is evaluated and defense strategy recommendations are generated to provide decision support for system operators.

Images