Substation intelligent terminal intrusion detection method and system based on distributed federal reinforcement learning

Abstract

The invention discloses a transformer substation intelligent terminal intrusion detection method and system based on distributed federal reinforcement learning. The method comprises the steps that historical intrusion detection data are collected, a meta-strategy is constructed through offline reinforcement learning, input of the meta-strategy comprises a current state and an attack action, output of the meta-strategy comprises a defense action, and the meta-strategy is rewarded or punished based on a reward function according to a defense result after the meta-strategy is implemented; defining a value function of a system state and an action based on the reward function, updating the value function based on a reinforcement learning method, and performing iterative training on the meta-strategy according to the optimized value function until the meta-strategy converges to obtain an intrusion detection model based on distributed federal reinforcement learning; and detecting a real-time security event of the transformer substation by using the intrusion detection model based on distributed federal reinforcement learning. According to the scheme of the invention, the overall security and stability of the network are improved.

Description

Technical Field

[0001] The present invention belongs to the field of network information security, and in particular relates to a substation intelligent terminal intrusion detection method and system based on distributed federated reinforcement learning. Background Art

[0002] With the continuous advancement of power intelligence construction, modern power systems are gradually controlled by intelligent terminals. Various business departments develop, deploy, and operate information systems with different functions to manage power production, distribution, control, and user data collection. These systems are usually connected to the Internet, and the stored data and services provided inevitably become targets of various potential attacks. Once the power information system of a substation is attacked, it may affect the normal operation of the power system and even have catastrophic consequences. Therefore, how to effectively prevent the power system from being invaded and attacked and ensure the safe operation of the power information system is particularly urgent.

[0003] Substations are usually distributed in various places in towns and villages, with large numbers and wide distribution. However, due to safety regulations, the equipment in the station needs to be monitored, operated and maintained. In actual operation, the maintenance and safety monitoring of substations require high accuracy and real-time performance, especially in preventing and detecting unauthorized intrusions. Traditional manual monitoring methods are not only time-consuming and labor-intensive, but also difficult to meet the needs of real-time response and repeated detection.

[0004] With the advancement of digitalization and intelligence of power systems, the construction of digital twins of power systems provides a way of remote control. The control system can remotely monitor and control the equipment in the substation through the network in the substation, and can continuously monitor the physical safety and operating status of the substation, which not only improves the monitoring efficiency, but also enhances the reaction speed, so that it can respond immediately when any abnormality or intrusion behavior is detected. However, at the same time, it also brings several problems. First, substations and remote control servers are often distributed far away, and there is a transmission delay between them. Some control commands with high real-time requirements are not responded in time; second, a large amount of data needs to be transmitted between substations and remote control servers, which not only consumes transmission resources, but also involves data privacy issues; third, there are many substations, and servers are difficult to meet computing needs; fourth, the robustness and fault tolerance of the existing system need to be improved. The existence of these problems has brought many challenges to the daily operation and emergency handling of substation systems. The application of federated reinforcement learning in power systems provides a new idea for the security protection of power systems. Federated reinforcement learning combines the advantages of federated learning and reinforcement learning. First, federated reinforcement learning data processing is done locally, and there is no need to upload local data to a remote control server, which effectively protects user privacy and reduces the risk of data leakage, which is especially important when processing sensitive information (such as power data). Secondly, federated reinforcement learning uses the computing resources of multiple devices or nodes for training, which improves the overall computing efficiency and reduces the computing burden of a single device. This is especially beneficial for reinforcement learning tasks that require a lot of computing resources. Through the diverse data on different devices or nodes, more generalized strategies and models can be learned to deal with various intrusion attacks. In addition, federated reinforcement learning also has high robustness and fault tolerance. Even if some devices or nodes fail, the system can continue to run and train. The distributed architecture makes the system less susceptible to single point failures. Summary of the invention

[0005] In order to address the deficiencies in the prior art, the present invention provides a substation intelligent terminal intrusion detection method and system based on distributed federated reinforcement learning to build an intelligent and adaptive network security defense system, and use reinforcement learning methods to dynamically adjust and optimize firewall rules to deal with various complex network attack behaviors.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions.

[0007] The present invention first discloses a substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning, which includes the following steps:

[0008] Step 1, collect historical intrusion detection data, encode the attack actions in the historical intrusion detection data into a graph representation of high-dimensional attack action data, use a graph convolutional network for learning, encode the high-dimensional attack action data into a low-dimensional attack action vector representation, and construct a meta-strategy through offline reinforcement learning. The input of the meta-strategy includes the current state and the low-dimensional attack action vector representation, and the output includes a defense action. According to the defense result after implementing the meta-strategy, the meta-strategy is rewarded or punished based on a reward function;

[0009] Step 2, defining the value function of the system state and action based on the reward function, updating the value function based on the reinforcement learning method, iteratively training the meta-strategy according to the optimized value function until the meta-strategy converges, and obtaining an intrusion detection model based on distributed federated reinforcement learning;

[0010] Step 3: Use the distributed federated reinforcement learning-based intrusion detection model to detect real-time security events in the substation.

[0011] The present invention further includes the following preferred embodiments:

[0012] The step of constructing a meta-strategy through offline reinforcement learning further includes:

[0013] Construct a security event model to store historical data of security events as security alerts for firewall policies, a dangerous event model to store data of events that were not successfully blocked, and a general experience pool to store all historical data that occurred.

[0014] The value function that defines the system state and actions further includes:

[0015] Value function Q(s, a, d):

[0016]

[0017] E a,d represents the expectation under actions a and d, Z u represents the maximum expected discounted reward and the maximum expected discounted reward that can be obtained from now to the future under the influence of the policy u currently executed by the firewall, where E u represents the expectation under strategy u, γ represents the discount factor affecting the future, and its value range is [0, 1]. The reward function M(s t , a t , d t ,s t+1 ) is used to measure the effect of the system after performing an action. Its input is the state s t , Attack Action a t , defensive action d t and the next state s t+1, the output is a real value m.

[0018] The iterative training of the meta-strategy according to the optimized value function further comprises:

[0019] By optimizing the strategy on the task distribution through the inner loop and optimizing the overall performance on the task through the outer loop, the meta-strategy is trained to achieve faster adaptation and optimization in new tasks:

[0020] θ * = argmax θ E T～p(T) [R(π θ )]

[0021]

[0022] Among them, θ is the parameter of the meta-strategy, θ * is the strategy optimization goal, by optimizing θ, maximizing the expected return, E T～p(T) is the expected value operation of the trajectory T, T is the task, p(T) is the task distribution, R is the reward, is the meta-training task T i The loss function on π meta represents the meta-strategy, β is the learning rate of the outer loop, and π i is the i-th strategy, ω i are the corresponding weights, which are dynamically adjusted based on the performance on different tasks.

[0023] The present invention also discloses a substation intelligent terminal intrusion detection system based on distributed federated reinforcement learning using the above-mentioned substation intelligent terminal intrusion detection method, including:

[0024] A construction module is used to collect historical intrusion detection data, encode attack actions in the historical intrusion detection data into a graph representation of high-dimensional attack action data, use a graph convolutional network for learning, encode the high-dimensional attack action data into a low-dimensional attack action vector representation, and construct a meta-strategy through offline reinforcement learning, wherein the input of the meta-strategy includes a current state and a low-dimensional attack action vector representation, and the output includes a defense action, and the meta-strategy is rewarded or punished based on a reward function according to the defense result after the implementation of the meta-strategy;

[0025] An optimization module, used for defining a value function of a system state and an action based on the reward function, updating the value function based on a reinforcement learning method, iteratively training the meta-strategy according to the optimized value function until the meta-strategy converges, and obtaining an intrusion detection model based on distributed federated reinforcement learning;

[0026] The detection module is used to detect real-time security events of substations using the intrusion detection model based on distributed federated reinforcement learning.

[0027] Accordingly, the present application also discloses a terminal, including a processor and a storage medium;

[0028] The storage medium is used to store instructions;

[0029] The processor is used to operate according to the instructions to execute the steps of the aforementioned substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning.

[0030] Correspondingly, the present application also discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the aforementioned substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning are implemented.

[0031] The beneficial effect of the present invention is that, compared with the prior art, the present invention provides a substation intelligent terminal intrusion detection method and system based on distributed federated reinforcement learning, which has strong adaptability. With the changes in the network environment and the emergence of new threats, the firewall can continuously optimize its own rules according to the optimal strategy obtained by the reinforcement learning method, thereby achieving efficient prediction and accurate blocking of network attack behaviors, improving the overall security and stability of the network, helping to reduce manual operation and maintenance costs, and having higher practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the substation intelligent terminal intrusion detection architecture based on distributed federated reinforcement learning in the present invention.

[0033] Figure 2 It is a flow chart of the substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning in the present invention.

[0034] Figure 3 It is a schematic diagram of the process of encoding high-dimensional attack action data into a low-dimensional vector representation of the attack action based on a graph convolutional network in the present invention.

[0035] Figure 4 It is a flow chart of the intelligent terminal experience model in the present invention.

[0036] Figure 5 It is a schematic diagram of the distributed training process in the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clear, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0038] The embodiments described in this application are only some embodiments of the present invention, not all embodiments. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the protection scope of the present invention.

[0039] The introduction of reinforcement learning methods can bring the following advantages:

[0040] 1. Intelligent decision-making: Through continuous learning and feedback, the reinforcement learning model can optimize the decision-making process. By continuously learning network traffic characteristics and business needs, it can adjust the priority strategy in real time to ensure priority processing of key businesses and select the best protection strategy to effectively respond to different types of network threats.

[0041] 2. Adaptability: On the one hand, reinforcement learning can realize automated and adaptive security strategies. Through continuous learning and adjustment, it can automatically adapt to new threat patterns and improve the protection effect. On the other hand, reinforcement learning has a strong threat detection capability. Through training models, reinforcement learning methods can identify complex attack patterns and abnormal behaviors, even unknown attacks. This capability greatly improves the detection accuracy and response speed of the firewall and reduces the possibility of false positives and false negatives.

[0042] 3. Sequential correlation: Reinforcement learning itself is an application paradigm for sequential decision-making tasks. Firewalls can better identify continuous and similar threats.

[0043] The present invention proposes a substation intelligent terminal intrusion detection method and system based on distributed federated reinforcement learning, which applies reinforcement learning theory to firewall strategy optimization and management. The system builds an intelligent and adaptive network security defense system, which uses reinforcement learning methods to dynamically adjust and optimize firewall rules to deal with various complex network attack behaviors.

[0044] First, the network environment is modeled as a Markov decision process. The action space is defined, which represents a series of actions that the firewall may take, including adding new access control rules, deleting or modifying existing rules, adjusting the priority of security policies, etc. These actions are used to achieve dynamic adjustment of network access control policies. When the firewall successfully identifies and blocks malicious traffic, the system gives positive rewards; conversely, if it fails to effectively block it, it gives negative penalties. Using reinforcement learning methods, the firewall continuously learns and accumulates experience in a series of continuous state-action cycles. In each iteration, the firewall selects the best action based on its current state and historical experience, updates the firewall rules, and obtains new feedback from the environment, thereby forming a closed-loop mechanism of self-improvement and evolution.

[0045] See also Figure 2 The invention discloses a substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning, comprising the following steps:

[0046] Step 1: Collect historical intrusion detection data, encode the attack actions in the historical intrusion detection data into a graph representation of high-dimensional attack action data, use a graph convolutional network for learning, encode the high-dimensional attack action data into a low-dimensional attack action vector representation, and construct a meta-strategy through offline reinforcement learning. The input of the meta-strategy includes the current state and the low-dimensional attack action vector representation, and the output includes defense actions. According to the defense result after implementing the meta-strategy, the meta-strategy is rewarded or punished based on the reward function.

[0047] Preferably, the attack action belongs to one or a combination of the following categories:

[0048] DoS attack, MITM attack, phishing attack, ransomware, password attack, injection attack, semantic URL attack, DNS spoofing, SYN attack or HTTP flood attack.

[0049] Among them, the DoS attack action is represented by a graph vector as [request type, request duration, request bandwidth, request frequency];

[0050] The MITIM attack action is represented by a graph vector as [communication type, interception point, data operation, response delay];

[0051] Phishing attack actions are represented by graph vectors as [attack vector, deception type, whether it contains malicious links, malicious link type];

[0052] The ransomware attack action is represented by a graph vector as [propagation method, encryption strength, attacked file type, ransom demand, unlock countdown];

[0053] Password attack actions are represented by graph vectors as [attempt frequency, password complexity, target service, automated tool];

[0054] The injection attack action is represented by a graph vector as [injection point, injection type, injection technique, injection encoding];

[0055] The semantic URL attack action is represented by a graph vector as [URL length, number of parameters, obfuscation technique, access frequency];

[0056] The DNS spoofing attack action is represented by a graph vector as [response tampering, target domain name, fake IP type, attack duration];

[0057] The SYN attack action is represented by a graph vector as [packet sending frequency, destination port, forged source IP, attack duration];

[0058] HTTP flood attack actions are represented by a graph vector as [request type, request frequency, target URL, payload size].

[0059] See also Figure 1 ,In this method, the following three models are constructed through offline ,reinforcement learning based on historical data: ,Security event model M1: stores historical data of security events, ,used as security alerts for firewall policies to improve the ,efficiency of firewall resisting similar historical threats; ,Danger event model M2, is used to store event data that ,was not successfully blocked to improve the defense coefficient; ,General experience pool M3, stores all historical data that ,appears, and is suitable for the first occurrence of security events.

[0060] In a round of time, each decision is a time step t. When an access occurs, a security event is recorded as e, a security event that is successfully identified and effectively blocked is E1, and a security event that is not successfully blocked is E2. All security events are E, the state space is recorded as S, and the action space A and D represent attack actions and defense actions respectively. The current strategy adopted by the firewall is recorded as u, its input is the state s and attack action a, and the output is the defense action d.

[0061] Reward function M(s t , a t , d t ,s t+1 ) is used to measure the effect of the system after performing an action. Its input is the state s t , Attack Action a t , defensive action d t and the next state s t+1 , the output is a real value m. In the present invention, a positive reward m>0 is given when malicious traffic is successfully blocked, and a negative reward m<0 is given when it fails to be effectively blocked. The reward function is defined as:

[0062]

[0063] Among them, r block is the basic positive reward for successfully blocking malicious traffic, r fail is the basic negative reward (usually negative) when malicious traffic is not effectively blocked, e block is the efficiency of successfully blocking malicious traffic, ranging between [0,1], c a and c d are the costs of attack and defense actions respectively.

[0064] Figure 3This is a schematic diagram of the process of encoding high-dimensional attack action data into low-dimensional vector representation of attack actions based on graph convolutional networks in the present invention. Encode the attack actions in historical intrusion detection data into graph representations, update the node representation by aggregating neighbor information, and effectively maintain the topological structure and node attributes of the original graph data; use graph convolutional networks for learning, encode high-dimensional attack action data into attack actions represented by low-dimensional vectors for meta-strategy construction.

[0065] Step 2: Based on the reward function, define the value function of the system state and action, update the value function based on the reinforcement learning method, iteratively train the meta-strategy according to the optimized value function until the meta-strategy converges, and obtain an intrusion detection model based on distributed federated reinforcement learning.

[0066] Define the value function Q(s, a, d):

[0067]

[0068] E a,d represents the expectation under actions a and d, Z u represents the maximum expected discounted reward and the maximum expected discounted reward that can be obtained from now to the future under the influence of the policy u currently executed by the firewall, where E u represents the expectation under strategy u, γ represents the discount factor that affects the future, and its value range is [0, 1].

[0069] First, we construct an offline dataset D = {(s t , a t , r t ,s t+1 )}, build a behavior model π that uses offline data to train β , the strategy model π' is trained by behavior cloning, and its loss function is:

[0070] L(π β )=E (s，a)～D [-logπ β (a|s)]

[0071] Where E (s，a)～D It represents the expected value of the state-action pair (s, a) under the data distribution D.

[0072] When optimizing the strategy, the behavior cloning term is added to the update strategy, and the loss function is:

[0073] L(π')=argmaxπ E s～D [Q(s,π(s),d)-α||π(s)-π β (s)||]

[0074] Among them, π(s) represents the action selected by strategy π in a given state s, α is a weight hyperparameter used to balance the reinforcement learning goal and the behavior cloning goal, and π β (s) is the behavior strategy, which is obtained by training offline data in offline learning.

[0075] Then update the Q value network, and the loss function is:

[0076] L(Q)=E (s，a，r，s')～D [(Q(s,a)-(r+γmin i＝1，2 Q'(s',π(s'),d'))) 2 ]

[0077] Among them, E (s，a，r，s')～D It represents the expected value calculation of all the four-tuples (s, a, r, s') in the experience replay buffer D, and Q'(s', π(s'), d') is the target Q value after selecting the action according to the strategy in the next state s'.

[0078] Repeatedly perform strategy optimization and value function update until the strategy converges. The three models are combined to form an intrusion detection model based on distributed federated reinforcement learning, and different models are selected for different intrusions.

[0079] In addition, in order to make the strategy better applicable to various scenarios, the strategy is optimized on the task distribution through the inner loop, and the overall performance on the task is optimized through the outer loop. A meta-strategy is trained to achieve faster adaptation and optimization in new tasks. The specific formula is:

[0080] θ * = argmax θ E T～p(T) [R(π θ )]

[0081]

[0082]

[0083]

[0084] Among them, θ is the parameter of the meta-strategy, θ * is the strategy optimization goal, by optimizing θ, maximizing the expected return, E T～p(T) is the expected value operation of the trajectory T, T is the task, p(T) is the task distribution, R is the reward, is the meta-training task T i The loss function on π meta represents the meta-strategy; β is the learning rate of the outer loop, π i is the i-th strategy, ω iare the corresponding weights, which can be dynamically adjusted based on the performance on different tasks.

[0085] For the value function, a neural network is used to fit the value function model of accurate prediction. The update of the value function is based on the principle of temporal difference to minimize the prediction difference.

[0086]

[0087] The loss function is:

[0088]

[0089]

[0090] Among them, α and β are weight parameters used to balance the loss, N is the total number of samples, and y t+1 is the target value, Q(s t , a t , d t ) is the value function estimate at the current moment, m t+1 is the immediate reward. By combining the differences between these two parts and weighting them by weight parameters α and β, the loss function can be more flexible to adapt to different reinforcement learning tasks.

[0091] Step 3: Use the distributed federated reinforcement learning-based intrusion detection model to detect real-time security events in the substation.

[0092] The beneficial effect of the present invention is that, compared with the prior art, the present invention provides a substation intelligent terminal intrusion detection method and system based on distributed federated reinforcement learning, which has strong adaptive capabilities. With the changes in the network environment and the emergence of new threats, the firewall can continuously optimize its own rules according to the optimal strategy obtained by the reinforcement learning method, thereby achieving efficient prediction and accurate blocking of network attack behaviors, improving the overall security and stability of the network, helping to reduce manual operation and maintenance costs, and having higher practicality.

[0093] The present invention may be a system, a method and / or a computer program product. The present invention also discloses a substation intelligent terminal intrusion detection system based on distributed federated reinforcement learning based on the aforementioned substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning, comprising:

[0094] A construction module is used to collect historical intrusion detection data, encode attack actions in the historical intrusion detection data into a graph representation of high-dimensional attack action data, use a graph convolutional network for learning, encode the high-dimensional attack action data into a low-dimensional attack action vector representation, and construct a meta-strategy through offline reinforcement learning, wherein the input of the meta-strategy includes a current state and a low-dimensional attack action vector representation, and the output includes a defense action, and the meta-strategy is rewarded or punished based on a reward function according to the defense result after the implementation of the meta-strategy;

[0095] An optimization module, used for defining a value function of a system state and an action based on the reward function, updating the value function based on a reinforcement learning method, iteratively training the meta-strategy according to the optimized value function until the meta-strategy converges, and obtaining an intrusion detection model based on distributed federated reinforcement learning;

[0096] The detection module is used to detect real-time security events of substations using the intrusion detection model based on distributed federated reinforcement learning.

[0097] Based on the spirit of the present invention, those skilled in the art can easily think that a computer program product can be obtained based on the aforementioned substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning. The computer program product may include a computer-readable storage medium, which carries computer-readable program instructions for enabling a processor to implement various aspects of the present disclosure. That is, the present application also includes a terminal, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the aforementioned substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning.

[0098] Computer readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device. Computer readable storage medium can be, for example, - but not limited to - electrical storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, for example, punch card or groove protrusion structure with instructions stored thereon and any suitable combination of the above. Computer readable storage medium used here is not interpreted as instantaneous signal itself, such as radio wave or other free propagating electromagnetic wave, electromagnetic wave propagated by waveguide or other transmission medium (for example, light pulse by optical fiber cable) or electrical signal transmitted by wire.

[0099] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0100] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages-such as Smalltalk, C++, etc., and conventional procedural programming languages-such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network-including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may be personalized by utilizing the state information of a computer-readable program instruction, and the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning, characterized in that: The following steps are involved: Step 1, collect historical intrusion detection data, encode the attack actions in the historical intrusion detection data into a graph representation of high-dimensional attack action data, use a graph convolutional network for learning, encode the high-dimensional attack action data into a low-dimensional attack action vector representation, and construct a meta-strategy through offline reinforcement learning. The input of the meta-strategy includes the current state and the low-dimensional attack action vector representation, and the output includes a defense action. According to the defense result after implementing the meta-strategy, the meta-strategy is rewarded or punished based on a reward function; Step 2, defining the value function of the system state and action based on the reward function, updating the value function based on the reinforcement learning method, iteratively training the meta-strategy according to the optimized value function until the meta-strategy converges, and obtaining an intrusion detection model based on distributed federated reinforcement learning; Step 3: Use the distributed federated reinforcement learning-based intrusion detection model to detect real-time security events in the substation.

2. The substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning according to claim 1 is characterized in that: The step of constructing a meta-strategy through offline reinforcement learning further includes: Construct a security event model to store historical data of security events as security alerts for firewall policies, a dangerous event model to store data of events that were not successfully blocked, and a general experience pool to store all historical data that occurred.

3. The substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning according to claim 2 is characterized in that: The value function of defining the system state and action based on the reward function further includes: Define the state value function Q(s, a, d): E a,d represents the expectation under actions a and d, Z u represents the maximum expected discounted reward and the maximum expected discounted reward that can be obtained from now to the future under the influence of the policy u currently executed by the firewall, where E u represents the expectation under strategy u, γ represents the discount factor affecting the future, and its value range is [0, 1]. The reward function M(s t , a t , d t ,s t+1 ) is used to measure the effect of the system after performing an action. Its input is the state s t , Attack Action a t , defensive action d t and the next state s t+1 , the output is a real value m.

4. The substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning according to claim 3 is characterized in that: The iterative training of the meta-strategy according to the optimized value function further comprises: By optimizing the strategy on the task distribution through the inner loop and optimizing the overall performance on the task through the outer loop, the meta-strategy is trained to achieve faster adaptation and optimization in new tasks: i * =argmax θ E T～p(T) [R(πθ)] Among them, θ is the parameter of the meta-strategy, θ * is the strategy optimization goal, by optimizing θ, maximizing the expected return, E T～p(T) is the expected value operation of the trajectory T, T is the task, p(T) is the task distribution, R is the reward, is the task T in the meta-training phase i The loss function on π meta represents the meta-strategy, β is the learning rate of the outer loop, and π i is the i-th strategy, ω i are the corresponding weights, which are dynamically adjusted based on the performance on different tasks.

5. A substation intelligent terminal intrusion detection system based on distributed federated reinforcement learning, characterized in that: include: A construction module is used to collect historical intrusion detection data, encode attack actions in the historical intrusion detection data into a graph representation of high-dimensional attack action data, use a graph convolutional network for learning, encode the high-dimensional attack action data into a low-dimensional attack action vector representation, and construct a meta-strategy through offline reinforcement learning, wherein the input of the meta-strategy includes a current state and a low-dimensional attack action vector representation, and the output includes a defense action, and the meta-strategy is rewarded or punished based on a reward function according to the defense result after the implementation of the meta-strategy; An optimization module, used for defining a value function of a system state and an action based on the reward function, updating the value function based on a reinforcement learning method, iteratively training the meta-strategy according to the optimized value function until the meta-strategy converges, and obtaining an intrusion detection model based on distributed federated reinforcement learning; The detection module is used to detect real-time security events of the substation using the intrusion detection model based on distributed federated reinforcement learning.

6. The substation intelligent terminal intrusion detection system based on distributed federated reinforcement learning according to claim 5 is characterized in that: The building block is further used for: Construct a security event model to store historical data of security events as security alerts for firewall policies, a dangerous event model to store data of events that were not successfully blocked, and a general experience pool to store all historical data that occurred.

7. The substation intelligent terminal intrusion detection system based on distributed federated reinforcement learning according to claim 6 is characterized in that: The optimization module is further used for: Define the state and value function Q(s, a, d): E a,d represents the expectation under actions a and d, Z u represents the maximum expected discounted reward and the maximum expected discounted reward that can be obtained from now to the future under the influence of the policy u currently executed by the firewall, where E u represents the expectation under strategy u, γ represents the discount factor affecting the future, and its value range is [0, 1]. The reward function M(s t , a t , d t ,s t+1 ) is used to measure the effect of the system after performing an action. Its input is the state s t , Attack Action a t , defensive action d t and the next state s t+1 , the output is a real value m.

8. The substation intelligent terminal intrusion detection system based on distributed federated reinforcement learning according to claim 7 is characterized in that: The optimization module is further used for: By optimizing the strategy on the task distribution through the inner loop and optimizing the overall performance on the task through the outer loop, the meta-strategy is trained to achieve faster adaptation and optimization in new tasks: i * =argmax θ E T～p(T) [R(π θ )] Among them, θ is the parameter of the meta-strategy, θ * is the strategy optimization goal, by optimizing θ, maximizing the expected return, E T～p(T) is the expected value operation of the trajectory T, T is the task, p(T) is the task distribution, R is the reward, is the task T in the meta-training phase i The loss function on π meta represents the meta-strategy, β is the learning rate of the outer loop, and π i is the i-th strategy, ω i are the corresponding weights, which are dynamically adjusted based on the performance on different tasks.

9. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is used to operate according to the instructions to execute the steps of the substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning according to any one of claims 1-4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the substation intelligent terminal intrusion detection method based on distributed federated reinforcement learning described in any one of claims 1 to 4 are implemented.

Images