Hidden attack system based on reinforcement learning and applied to industrial control system

Abstract

The invention provides a hidden attack system based on reinforcement learning applied to an industrial control system, which is used for executing hidden attack on the industrial control system and comprises a data acquisition module, a data processing module and a hidden attack processing module, environment data and operation data of a current time window sent by each programmable logic controller to the data acquisition and monitoring control system are intercepted in a preset cheating mode; the intelligent agent is used for adding disturbance to the environment data and the operation data of each current time window so as to obtain the environment data and the operation data of each tampered current time window; and the data transmission module is used for sending all tampered environment data and operation data of the current time window to the data acquisition and monitoring control system. According to the technical scheme, the intelligent agent is obtained based on reinforcement learning to tamper the data of the sensor and the actuator of the physical layer of the industrial control system, the learning efficiency can be improved, and the hidden attack success rate can also be improved.

Description

Technical Field

[0001] The present invention relates to the field of industrial control system security protection, specifically to a covert attack technology in the field of industrial control system security protection, and more specifically to a covert attack system based on reinforcement learning applied to industrial control systems. Background Art

[0002] Industrial control systems are complex systems that monitor and control industrial production processes. They are mainly composed of computer equipment, communication networks, and physical devices such as sensors and actuators. Industrial control systems are widely used in various industrial fields, such as manufacturing, electricity, chemicals, oil and gas, water treatment, transportation, etc. Due to the important position of industrial control systems and their own vulnerability, they are often threatened by attacks from all sides and have serious security risks. In order to eliminate attack threats and security risks and ensure the normal and orderly operation of industrial control systems, practitioners use anomaly detection systems to detect anomalies and block possible attack behaviors in time. Among them, the anomaly detection system based on process data realizes the function of anomaly detection by modeling or learning the spatiotemporal dependency of sensor and actuator values.

[0003] However, there is a class of adversarial attacks, namely covert attacks, which evade detection by changing the inputs (sensor and actuator values) of anomaly detection systems, thereby masking the anomalies occurring in the underlying physical process. In order to evaluate and improve the resistance of anomaly detection systems to covert attacks, effective covert attack methods need to be designed.

[0004] According to the attacker's knowledge of the target industrial control system, covert attacks can be divided into black-box attacks and white-box attacks. In the black-box scenario, the attacker does not know the implementation principle and parameter details of the anomaly detection system; in the white-box scenario, the attacker has detailed knowledge of the implementation principle and parameter details of the anomaly detection system. According to the attacker's attack capabilities, covert attacks can also be divided into restricted covert attacks and unrestricted covert attacks. In the case of restricted covert attacks, the attacker can only manipulate the values ​​of some sensors and actuators; while in the case of unrestricted covert attacks, the attacker can arbitrarily manipulate the values ​​of all sensors and actuators. Therefore, unrestricted covert attacks in white-box scenarios are the most ideal form of covert attacks.

[0005] Existing covert attack methods for industrial control systems can be divided into three categories according to the attack principles: replay attacks, iteration-based attacks, and learning-based attacks.

[0006] Among them, the replay attack replaces the current values ​​with the historical values ​​of the normal operation of sensors and actuators. This attack method can achieve good results in unrestricted situations, but the attack effect is very poor in restricted covert attack situations.

[0007] Iteration-based attacks are generally used in white-box scenarios. According to the implementation principle and parameter details of the anomaly detection system, the attacker uses the coordinate descent method and other iterations to find appropriate perturbations of the sensor and actuator values ​​to evade the detection of the anomaly detection system. However, the execution time of this attack method is uncertain, and it may not be possible to find a feasible solution within the specified time. In addition, the assumptions of the white-box scenario are too strong and difficult to achieve in reality.

[0008] Learning-based attacks are generally used in black box scenarios. They use autoencoders and other techniques to learn the spatiotemporal dependencies between sensors and actuators from historical data of normal operation of industrial control systems, and then generate normal values ​​that will not be recognized by the anomaly detection system based on the current abnormal values ​​of sensors and actuators. However, this method requires long-term monitoring of industrial control systems to obtain sufficient training data, and existing methods have poor attack effects in restricted covert attack scenarios.

[0009] In summary, although the existing covert attack methods can realize covert attacks in black-box scenarios and white-box scenarios, the attack effects of the existing covert attack methods in restricted scenarios in black-box scenarios are poor, and it is difficult to effectively evaluate the resistance of anomaly detection systems to covert attacks.

[0010] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art. Summary of the invention

[0011] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a covert attack system based on reinforcement learning applied to industrial control systems and an anomaly detection and evaluation method.

[0012] The objectives of the present invention are achieved through the following technical solutions.

[0013] According to a first aspect of the present invention, there is provided a covert attack system based on reinforcement learning applied to an industrial control system, which is used to perform covert attacks on the industrial control system. The industrial control system includes a physical layer and a monitoring and control layer, wherein the physical layer includes a plurality of sensors and a plurality of actuators, each actuator is used to execute operation instructions and record operation data, and each sensor is used to monitor the environmental data of the industrial control system; the monitoring and control layer includes a data acquisition and monitoring control system and a plurality of programmable logic controllers, each programmable logic controller is used to receive the environmental data sent by the corresponding sensor and the operation data sent by the corresponding actuator, and adjust the behavior of the corresponding actuator based on its own pre-configured control logic; the data acquisition and monitoring control system is used to receive the environmental data and operation data sent by all programmable logic controllers; the covert attack system includes: a data acquisition module, which is used to obtain the environmental data and operation data sent by the corresponding programmable logic controllers; When the industrial control system is attacked, the environment data and operation data of the current time window sent by each programmable logic controller to the data acquisition and monitoring control system are intercepted in a preset deceptive manner; the intelligent agent is used to add disturbances to the environment data and operation data of each current time window to obtain the environment data and operation data of each tampered current time window; wherein the intelligent agent uses the environment data and operation data of the current time window, the environment data and operation data of the historical time window, and the known attackable physical layer data as state input, and the tampered environment data and operation data of the current time window as action output, and uses a preset reinforcement learning method to execute multiple rounds of iterative learning to obtain a deep neural network model; the data transmission module is used to send all the tampered environment data and operation data of the current time window to the data acquisition and monitoring control system.

[0014] In some embodiments of the present invention, the preset spoofing means include: MAC address spoofing, DHCP address spoofing and ARP spoofing.

[0015] In some embodiments of the present invention, the intelligent agent uses the environmental data and operation data of the current time window, the environmental data and operation data of the historical time window, and the known attackable physical layer data as state input, and the tampered environmental data and operation data of the current time window as action output, and uses a preset reinforcement learning method to perform multiple rounds of iterative learning to obtain a deep neural network model, wherein multiple steps of iterative updates are performed in each round of iterative learning.

[0016] In some embodiments of the present invention, the industrial control system also includes a detection layer for receiving environmental data and operation data sent by the data acquisition and monitoring control system and performing anomaly detection. In each round of iterative learning, each iterative update process is: obtaining the environmental data of the current time window that each sensor in the physical layer wants to send to the corresponding programmable logic controller, and the operation data of the current time window that each actuator in the physical layer wants to send to the corresponding programmable logic controller; using the environmental data and operation data of all current time windows, the environmental data and operation data of historical time windows, and the known attackable physical layer data as state input to obtain the environmental data and operation data of each current time window after tampering; using each current time window after tampering The environmental data and operation data of the previous time window are sent to the corresponding programmable logic controller, so that each programmable logic controller sends the corresponding tampered environmental data and operation data of the current time window to the data acquisition and monitoring control system, and waits for the data acquisition and monitoring control system to send the tampered environmental data and operation data of all current time windows to the detection layer for anomaly detection to obtain the detection result; based on the environmental data and operation data of all current time windows, the environmental data and operation data of historical time windows, the known attackable physical layer data, the tampered environmental data and operation data of all current time windows and the detection results, the preset reinforcement learning method is used to update the parameters of the deep neural network model constructed based on the reinforcement learning method.

[0017] In some embodiments of the present invention, the preset reinforcement learning is: a deep deterministic policy gradient method, an advantage actor-critic method, an asynchronous advantage actor-critic method, a trust region policy optimization method or a proximal policy optimization method.

[0018] According to a second aspect of the present invention, there is provided an anomaly detection evaluation method for evaluating the anomaly detection capability of an industrial control system when facing a covert attack. The method comprises: step S1, using the system as described in the first aspect of the present invention to perform a covert attack on the industrial control system; step S2, obtaining the detection result obtained by the detection layer in the industrial control system, when the detection result is no abnormality, the anomaly detection fails; otherwise, the anomaly detection is successful.

[0019] In some embodiments of the present invention, the detection layer is configured with an anomaly detector for detecting whether there are anomalies in the environmental data and operation data sent by the data acquisition and monitoring control system.

[0020] In some embodiments of the present invention, the anomaly detector is configured to perform anomaly detection on the environmental data and the operational data based on a residual approach or an invariant approach.

[0021] Compared with the prior art, the advantages of the present invention are: (1) the introduction of an intelligent agent based on reinforcement learning, which directly interacts and learns with the industrial control system through reinforcement learning, can save the intermediate links of operating data collection and improve learning efficiency; (2) in addition, the intelligent agent can simultaneously learn the operating logic and anomaly detection logic of the industrial control system. Compared with the existing single learning of the operating logic of the industrial control system or single learning of the anomaly detection logic, the learning of the intelligent agent is more comprehensive and can improve the success rate of covert attacks; (3) the input of the intelligent agent includes known attackable physical layer data set as a mask vector, which can improve the richness of the intelligent agent's learning experience and make the intelligent agent suitable for different restricted scenarios; (4) the covert attack system based on reinforcement learning can be applied to any anomaly detector based on process data, broadening the scope of application of covert attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0023] Figure 1 A schematic diagram of a covert attack system according to an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of an agent iterative learning process according to an embodiment of the present invention;

[0025] Figure 3 Schematic diagram of an example of anomaly detection evaluation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] As mentioned in the background technology section, although existing covert attack methods can implement covert attacks in black box scenarios and white box scenarios, the attack effects of existing covert attack methods in restricted scenarios in black box scenarios are poor, and it is difficult to effectively evaluate the ability of anomaly detection systems to resist covert attacks.

[0028] In order to solve the above problems, the inventor studied the existing learning-based covert attack method and found that the learning-based covert attack method requires long-term monitoring and acquisition of the operation data of the industrial control system, which will lead to two problems: first, long-term data collection is becoming more and more difficult, and second, it is impossible to determine the appropriate amount of data, and insufficient data will reduce the learning effect and thus reduce the efficiency of the covert attack. Based on the analysis of the existing learning-based covert attack method, the inventor proposed a new covert attack system, in which an intelligent agent is set to tamper with the data of the sensors and actuators of the physical layer of the industrial control system, wherein the intelligent agent directly interacts with the industrial control system through reinforcement learning, which can save the intermediate links of operation data collection and improve learning efficiency; not only that, the intelligent agent can also learn the operation logic and anomaly detection logic of the industrial control system at the same time. Compared with the existing single learning of the operation logic of the industrial control system or the single learning of the anomaly detection logic, the intelligent agent's learning is more sufficient, which can improve the success rate of the covert attack.

[0029] In summary, if Figure 1 As shown, the present invention provides a covert attack system based on reinforcement learning applied to an industrial control system, which is used to perform covert attacks on the industrial control system. The industrial control system includes a physical layer and a monitoring and control layer, wherein the physical layer includes multiple sensors and multiple actuators, each actuator is used to execute operation instructions and record operation data, and each sensor is used to monitor the environmental data of the industrial control system; the monitoring and control layer includes a data acquisition and monitoring control system and multiple programmable logic controllers, each programmable logic controller is used to receive the environmental data sent by the corresponding sensor and the operation data sent by the corresponding actuator, and adjust the behavior of the corresponding actuator based on its own pre-configured control logic; the data acquisition and monitoring control system is used to receive the environmental data and operation data sent by all programmable logic controllers; the covert attack system includes: a data acquisition module, which is used to obtain the environmental data and operation data sent by all programmable logic controllers in the data acquisition and monitoring control system; When the industrial control system is attacked, the environmental data and operation data of the current time window sent by each programmable logic controller to the data acquisition and monitoring control system are intercepted in a preset deceptive manner; the intelligent agent is used to add disturbances to the environmental data and operation data of each current time window to obtain the environmental data and operation data of each tampered current time window; wherein the intelligent agent uses the environmental data and operation data of the current time window, the environmental data and operation data of the historical time window, and the known attackable physical layer data as state input, and the tampered environmental data and operation data of the current time window as action output, and uses a preset reinforcement learning method to execute multiple rounds of iterative learning to obtain a deep neural network model; the data transmission module is used to send all the tampered environmental data and operation data of the current time window to the data acquisition and monitoring control system.

[0030] Before specifically introducing the embodiments of the present invention, a covert attack method against an industrial control system is briefly introduced.

[0031] A covert attack is when an attacker uses technical means to intercept the physical layer data transmitted by the physical layer of an industrial control system, add disturbances to the physical layer data, and then send it back to the industrial control system, making it impossible for the industrial control system to detect anomalies in the physical layer data after the disturbance is added, thereby achieving the purpose of a covert attack.

[0032] Specifically, the above covert attack process can be formalized as follows.

[0033] The industrial control system should satisfy the following condition for anomaly detection of physical layer data (data from sensors and actuators) transmitted by the physical layer: F(X) = 1, where F(·) represents the anomaly detection function and F(X) = 1 indicates that an anomaly exists in the underlying physical layer. The purpose of covert attack is to find X ′ =X+ΔX, so that F(X ′ )=0, that is, adding disturbance ΔX to the physical layer data to obtain X ′ , which results in the industrial control system being unable to detect the anomaly that is occurring.

[0034] In order to better understand the present invention, each component of the covert attack system is described in detail below in conjunction with specific embodiments.

[0035] 1. Data Acquisition Module

[0036] The data acquisition module is used to intercept the environmental data and operation data of the current time window sent by each programmable logic controller to the data acquisition and monitoring control system in a preset deceptive manner when the industrial control system is attacked.

[0037] According to an embodiment of the present invention, the preset spoofing means include: MAC address spoofing, DHCP address spoofing and ARP spoofing.

[0038] It should be noted that the industrial control system can be an industrial control system actually running in the real world, or it can be a simulated industrial control system built using simulation software or simulation system according to the type and configuration information of the industrial control system. For example, open source platforms provided by research institutions such as CISPA's water treatment simulation platform DHALSIM, and commercial simulation software provided by manufacturers such as Siemens' Simcenter series simulation software. It should also be noted that if a covert attack is launched by an adversary, the configuration information of the industrial control system may come from an attacker lurking inside the industrial control system, or intelligence leakage caused by insecure communications of the owner of the industrial control system; if the covert attack is actively initiated by the owner of the industrial control system to evaluate the current anomaly detection system's resistance to covert attacks, then the configuration information of the industrial control system can be directly provided by the industrial control system.

[0039] 2. Intelligent Agent

[0040] The intelligent agent is used to add disturbances to the environmental data and operation data of each current time window to obtain the environmental data and operation data of each tampered current time window; wherein the intelligent agent uses the environmental data and operation data of the current time window, the environmental data and operation data of the historical time window, and the known attackable physical layer data as state input, and the tampered environmental data and operation data of the current time window as action output, and uses a preset reinforcement learning method to execute multiple rounds of iterative learning to obtain a deep neural network model.

[0041] According to one embodiment of the present invention, the intelligent agent uses the environmental data and operation data of the current time window, the environmental data and operation data of the historical time window, and the known attackable physical layer data as state input, and the tampered environmental data and operation data of the current time window as action output, and uses a preset reinforcement learning method to perform multiple rounds of iterative learning to obtain a deep neural network model, wherein multiple steps of iterative updates are performed in each round of iterative learning.

[0042] According to one embodiment of the present invention, the industrial control system also includes a detection layer for receiving environmental data and operation data sent by the data acquisition and monitoring control system and performing anomaly detection. In each round of iterative learning, each iterative update process is as follows: obtaining environmental data of the current time window that each sensor in the physical layer wants to send to the corresponding programmable logic controller, and operation data of the current time window that each actuator in the physical layer wants to send to the corresponding programmable logic controller; using the environmental data and operation data of all current time windows, environmental data and operation data of historical time windows, and known attackable physical layer data as state inputs to obtain environmental data and operation data of each current time window after tampering; using each current time window after tampering as state inputs to obtain environmental data and operation data of each current time window after tampering; The environmental data and operation data of the previous time window are sent to the corresponding programmable logic controller, so that each programmable logic controller sends the corresponding tampered environmental data and operation data of the current time window to the data acquisition and monitoring control system, and waits for the data acquisition and monitoring control system to send the tampered environmental data and operation data of all current time windows to the detection layer for anomaly detection to obtain the detection result; based on the environmental data and operation data of all current time windows, the environmental data and operation data of historical time windows, the known attackable physical layer data, the tampered environmental data and operation data of all current time windows and the detection results, the preset reinforcement learning method is used to update the parameters of the deep neural network model constructed based on the reinforcement learning method.

[0043] According to one embodiment of the present invention, the preset reinforcement learning is: a deep deterministic policy gradient method, an advantage actor-critic method, an asynchronous advantage actor-critic method, a trust region policy optimization method or a proximal policy optimization method.

[0044] In order to have a deeper understanding of the iterative learning process of the intelligent agent, a brief introduction is given by taking the deep deterministic policy gradient method as an example. Other reinforcement learning methods, like the deep deterministic policy gradient method, are relatively mature reinforcement learning technologies, so they will not be elaborated here one by one.

[0045] like Figure 2 As shown, the agent interacts with the industrial control system and performs multiple rounds of iterative learning based on a preset reinforcement learning method to obtain a deep neural network model. Among them, through multiple rounds of iterative learning, the agent can fully learn the operating logic of the industrial control system and the abnormal detection logic of the detection layer, and then add appropriate disturbances based on the abnormal values ​​of sensors and actuators to evade detection at the detection layer. Among them, multiple steps of iterative updates are performed in each round of iterative learning, and the process of each step of iterative update is as follows.

[0046] First, obtain the environmental data of the current time window that each sensor in the physical layer of the industrial control system wants to send to the corresponding programmable logic controller, and the operation data of the current time window that each actuator in the physical layer wants to send to the corresponding programmable logic controller. These data represent the operating status of the underlying physical layer. These data may be normal or abnormal. The obtained data is represented as a numerical matrix V = (X n-1 ,X n-2 ,X n-3 ,…,X 0 ), where X 0 Represents the environmental data and operation data of the current time window, X n-1 ,X n-2 ,X n-3 Represents the environmental data and operation data of the historical time window. The numerical matrix V records the data of n time windows in total. It should be noted that the environmental data and operation data of the historical time window are obtained from the physical layer of the industrial control system in the previous time window. It should be noted that since training requires a lot of interaction, and the real-world industrial control system cannot afford the cost of training, in the iterative learning process, the intelligent agent interacts with the simulated industrial control system to learn the operation logic of the industrial control system and the anomaly detection logic of the detection layer.

[0047] Then determine the attackable physical layer data, that is, the sensors and actuators that the attacker can tamper with, and represent them as a binary mask vector M. Among them, the mask vector is a method of selectively operating or filtering the corresponding bits in the original vector by setting the bits in the binary vector. Specifically, each bit in the mask vector corresponds to a sensor or actuator. When a bit in the mask vector is 1, it means that the sensor or actuator corresponding to the bit can be tampered with; when a bit in the mask vector is 0, it means that the sensor or actuator corresponding to the bit cannot be tampered with. It should be noted that the mask vector M can be adjusted according to the needs of attack or iterative learning.

[0048] Then, the numerical matrix V and the mask vector M constitute the state input S = (V, M) of the agent. The agent makes a decision based on the state input S, that is, the environmental data and operation data X in the current time window 0 Add a disturbance ΔX 0 Get the tampered data X 0 ′ =X 0 +ΔX 0 .

[0049] Then the tampered data X 0 ′ =X 0 +ΔX 0The data is sent to the corresponding programmable logic controller (PLC), so that each programmable logic controller sends the corresponding tampered environmental data and operation data of the current time window to the data acquisition and monitoring control system (SCADA), and waits for the data acquisition and monitoring control system to send the tampered environmental data and operation data of all current time windows to the detection layer for abnormal detection to obtain the detection result; at the same time, after receiving the tampered data, each programmable logic controller adjusts the behavior of the actuator based on its own pre-configured control logic, thereby changing the physical layer state of the industrial control system to obtain the state input S for the next iterative learning. ′ It should be noted that the control logic is written in the PLC in the form of code to control the actuator. For example, the control logic is to close the valve when the ambient temperature is greater than or equal to 80 degrees, or to turn on the motor when the ambient humidity is greater than or equal to a preset threshold.

[0050] Finally, based on the environmental data and operation data of all current time windows, the environmental data and operation data of historical time windows, the known attackable physical layer data, the environmental data and operation data of all current time windows after tampering, and the detection results, the preset reinforcement learning method is used to update the parameters of the deep neural network model constructed based on the reinforcement learning method. It should be noted that if the anomaly detection result is anomaly or the maximum number of training steps is reached, the current round of training is terminated; if the anomaly detection result is no anomaly (normal), the agent will receive a certain reward (reward), which is set to a positive constant to encourage the agent to conduct more steps of iterative learning with the industrial control system, and update the agent according to the update strategy of the deep deterministic policy gradient method according to <state, action, reward, new state> to complete this iterative learning. It should also be noted that in the black box scenario, the attacker cannot obtain the anomaly detection principle in the industrial control system. Therefore, when constructing the agent, the detection layer in the simulated industrial control system can be set to a combination of residual-based and invariant-based methods for anomaly detection, which can improve the robustness of the constructed agent and broaden its scope of application.

[0051] It should be noted that, through Figure 2 In the iterative learning process shown, the intelligent agent can learn the operating logic and anomaly detection logic of the industrial control system at the same time. Compared with the existing method of learning the operating logic of the industrial control system or learning the anomaly detection logic alone, the intelligent agent's learning is more comprehensive and can improve the success rate of covert attacks.

[0052] 3. Data transmission module

[0053] The data transmission module is used to send all the tampered environmental data and operation data of the current time window to the data acquisition and monitoring control system.

[0054] Based on the covert attack system described in the above embodiment, the present invention also proposes an anomaly detection evaluation method for evaluating the anomaly detection capability of an industrial control system when facing a covert attack, the method comprising: step S1, using the covert attack system described in the above embodiment to perform a covert attack on the industrial control system; step S2, obtaining the detection result obtained by the detection layer in the industrial control system, when the detection result is no abnormality, the anomaly detection fails; otherwise, the anomaly detection succeeds. It should be noted that in the anomaly detection evaluation process, what is evaluated is the anomaly detection capability of the real-world industrial control system when facing a covert attack.

[0055] According to an embodiment of the present invention, the detection layer is configured with an anomaly detector for detecting whether there are anomalies in the environmental data and operation data sent by the data acquisition and monitoring control system.

[0056] According to one embodiment of the present invention, the anomaly detector is configured to perform anomaly detection on environmental data and operation data based on a residual method or an invariant method. Among them, the residual-based method is to train the detection model by minimizing the loss function between the expected value and the observed value or between the input and the output (generally using the mean square error), and when the difference exceeds the set threshold, an abnormal alarm will be triggered; this abnormality detection method includes autoregressive (AR, AutoRegressive), autoencoder (AE, Auto Encoder) and other deep neural network (DNN, Deep NeuralNetwork) models. The invariant-based method is to identify anomalies by constructing a set of rules that are always satisfied when the industrial control system is running normally. The construction of rules often needs to be based on detailed knowledge of a specific industrial control system. Once a rule is not satisfied, it means that an anomaly has occurred; this detection method includes SFIG (Systematic Framework for Invariant Generation). It should be noted that the anomaly detection method configured by the anomaly detector in the real-world industrial control system is flexibly configured according to actual needs, and the present invention does not impose special restrictions.

[0057] In order to better understand the execution process of the anomaly detection evaluation method, Figure 3 The anomaly detection evaluation process shown in is taken as an example to illustrate. Figure 3It can be seen that a preset deception method is first used to intercept the environmental data and operation data of the current time window sent by each programmable logic controller to the data acquisition and monitoring control system; then, an intelligent agent is used to tamper with the intercepted environmental data and operation data of the current time window; then, the tampered environmental data and operation data of the current time window are sent to the data acquisition and monitoring control system, so that the data acquisition and monitoring control system sends the tampered environmental data and operation data of all current time windows to the detection layer for anomaly detection; finally, the detection result is obtained. When the detection result is normal (no anomaly), it indicates that the anomaly detection has failed.

[0058] The beneficial effects of the present invention are as follows: (1) an intelligent agent based on reinforcement learning is introduced, which directly interacts and learns with the industrial control system through reinforcement learning, which can save the intermediate links of operating data collection and improve learning efficiency; (2) the intelligent agent can simultaneously learn the operating logic and anomaly detection logic of the industrial control system. Compared with the existing method of learning the operating logic of the industrial control system or learning the anomaly detection logic alone, the intelligent agent's learning is more comprehensive and can improve the success rate of covert attacks; (3) the input of the intelligent agent includes known attackable physical layer data set as a mask vector, which can improve the richness of the intelligent agent's learning experience and make the intelligent agent suitable for different restricted scenarios; (4) the covert attack system based on reinforcement learning can be applied to any anomaly detector based on process data, which broadens the scope of application of covert attacks.

[0059] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0060] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0061] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0062] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A covert attack system based on reinforcement learning applied to an industrial control system, for performing covert attacks on the industrial control system, wherein the industrial control system includes a physical layer, a monitoring and control layer, wherein: The physical layer includes multiple sensors and multiple actuators. Each actuator is used to execute operation instructions and record operation data. Each sensor is used to monitor the environmental data of the industrial control system. The monitoring and control layer includes a data acquisition and monitoring control system and multiple programmable logic controllers. Each programmable logic controller is used to receive the environmental data sent by the corresponding sensor and the operation data sent by the corresponding actuator, and adjust the behavior of the corresponding actuator based on its own pre-configured control logic. The data acquisition and monitoring control system is used to receive environmental data and operation data sent by all programmable logic controllers; characterized in that the covert attack system includes: A data acquisition module is used to intercept the environmental data and operation data of the current time window sent by each programmable logic controller to the data acquisition and monitoring control system in a preset deceptive manner when the industrial control system is attacked; An intelligent agent is used to add disturbances to the environmental data and operation data of each current time window to obtain the environmental data and operation data of each tampered current time window; wherein the intelligent agent uses the environmental data and operation data of the current time window, the environmental data and operation data of the historical time window, and the known attackable physical layer data as state inputs, and the tampered environmental data and operation data of the current time window as action outputs, and uses a preset reinforcement learning method to execute multiple rounds of iterative learning to obtain a deep neural network model; The data transmission module is used to send all the tampered environmental data and operation data of the current time window to the data acquisition and monitoring control system.

2. The system according to claim 1, characterized in that The preset deception means include: MAC address deception method, DHCP address deception method and ARP deception method.

3. The system according to claim 2, characterized in that The intelligent agent uses the environmental data and operation data of the current time window, the environmental data and operation data of the historical time window, and the known attackable physical layer data as state input, and the tampered environmental data and operation data of the current time window as action output, and uses a preset reinforcement learning method to perform multiple rounds of iterative learning to obtain a deep neural network model, wherein multiple steps of iterative updates are performed in each round of iterative learning.

4. The system according to claim 3, characterized in that The industrial control system also includes a detection layer, which is used to receive environmental data and operation data sent by the data acquisition and monitoring control system and perform anomaly detection. In each round of iterative learning, the iterative update process of each step is: Acquire the environmental data of the current time window that each sensor in the physical layer intends to send to the corresponding programmable logic controller, and the operation data of the current time window that each actuator in the physical layer intends to send to the corresponding programmable logic controller; Using all acquired environment data and operation data of the current time window, environment data and operation data of the historical time window, and known attackable physical layer data as state input to obtain the tampered environment data and operation data of each current time window; Send the tampered environmental data and operation data of each current time window to the corresponding programmable logic controller, so that each programmable logic controller sends the corresponding tampered environmental data and operation data of the current time window to the data acquisition and monitoring control system, and wait for the data acquisition and monitoring control system to send the tampered environmental data and operation data of all current time windows to the detection layer for abnormality detection to obtain the detection result; Based on the environmental data and operation data of all current time windows, the environmental data and operation data of historical time windows, the known attackable physical layer data, the tampered environmental data and operation data of all current time windows and the detection results, a preset reinforcement learning method is used to update the parameters of the deep neural network model built based on the reinforcement learning method.

5. The system according to claim 4, characterized in that The default reinforcement learning is: deep deterministic policy gradient method, advantage actor-critic method, asynchronous advantage actor-critic method, trust region policy optimization method, or proximal policy optimization method.

6. An anomaly detection evaluation method for evaluating the anomaly detection capability of an industrial control system facing a covert attack, characterized in that: The method comprises: Step S1, using the system as described in any one of claims 1 to 5 to perform a covert attack on an industrial control system; Step S2: Obtain the detection result obtained by the detection layer in the industrial control system. When the detection result is no abnormality, the abnormality detection fails; otherwise, the abnormality detection succeeds.

7. The method according to claim 6, characterized in that The detection layer is equipped with an anomaly detector for detecting whether there are anomalies in the environmental data and operation data sent by the data acquisition and monitoring control system.

8. The method according to claim 7, characterized in that The anomaly detector is configured to perform anomaly detection on the environment data and the operation data in a residual-based manner or an invariant-based manner.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any method described in claims 6-8.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of any one of claims 6-8 by executing the executable instructions.

Images