A reinforcement learning based covert attack system applied to an industrial control system

Abstract

The application provides a hidden attack system based on reinforcement learning applied to an industrial control system, which is used for performing a hidden attack on the industrial control system, and the hidden attack system comprises a data acquisition module, which is used for intercepting environment data and operation data of each programmable logic controller in a current time window sent to a data acquisition and monitoring control system in a preset cheating manner when the industrial control system is attacked; an agent, which is used for adding disturbance to the environment data and the operation data of each current time window to obtain tampered environment data and operation data of each current time window; and a data transmission module, which is used for sending the tampered environment data and operation data of all current time windows to the data acquisition and monitoring control system. The technical scheme of the application tampers the data of sensors and actuators of a physical layer of the industrial control system by obtaining the agent based on reinforcement learning, so that the learning efficiency can be improved, and the success rate of the hidden attack can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of industrial control system security protection, in particular to the stealth attack technology in the field of industrial control system security protection, more particularly to a stealth attack system based on reinforcement learning applied to an industrial control system. BACKGROUND

[0002] Industrial control systems are complex systems that monitor and control industrial production processes, mainly composed of computer devices, communication networks, and physical devices such as sensors and actuators. Industrial control systems are widely used in various industrial fields, such as manufacturing, power, chemical, oil and gas, water treatment, transportation, etc. Due to the important position of industrial control systems and their own vulnerability, they are often threatened by attacks from all sides, and there are serious security risks. In order to eliminate attack threats and security risks and ensure the normal and orderly operation of industrial control systems, practitioners use anomaly detection systems to detect anomalies and block possible attack behavior in a timely manner. Among them, the anomaly detection system based on process data achieves the function of anomaly detection by modeling or learning the spatio-temporal dependence of sensor and actuator values.

[0003] However, there is a type of attack against attacks, namely stealth attacks, which evade detection by changing the input of the anomaly detection system (sensor and actuator values) to mask the anomalies occurring in the underlying physical process. In order to evaluate and improve the resistance of the anomaly detection system to stealth attacks, it is necessary to design effective stealth attack methods.

[0004] According to the attacker's knowledge of the target industrial control system, stealth attacks can be divided into black box attacks and white box attacks. In the black box scenario, the attacker does not know the implementation principle and parameter details of the anomaly detection system; in the white box scenario, the attacker has detailed knowledge of the implementation principle and parameter details of the anomaly detection system. According to the attack ability of the attacker, stealth attacks can also be divided into limited stealth attacks and unrestricted stealth attacks. In the case of limited stealth attacks, the attacker can only manipulate the values of some sensors and actuators; while in the case of unrestricted stealth attacks, the attacker can manipulate the values of all sensors and actuators arbitrarily. Therefore, unrestricted stealth attacks in the white box scenario are the most ideal form of stealth attacks.

[0005] Existing industrial control system stealth attack methods can be divided into three categories according to the attack principle: replay attacks, iteration-based attacks, and learning-based attacks.

[0006] Among them, the replay attack replaces the current values of sensors and actuators with historical values of normal operation, which can achieve good results in the case of unrestricted attacks, but the attack effect is poor in the case of limited stealth attacks.

[0007] The iteration-based attack is generally used in the white-box scenario, and the attacker finds suitable perturbations of the sensor and actuator values by iteration such as coordinate descent method to escape the detection of the anomaly detection system according to the implementation principle and parameter details of the anomaly detection system, but the execution time of the attack mode is uncertain, and a feasible solution may not be found within a specified time, and the assumption of the white-box scenario is too strong and is difficult to realize in reality.

[0008] The learning-based attack is generally used in the black-box scenario, and the spatio-temporal dependency between the sensor and the actuator is learned from the historical data of the normal operation of the industrial control system by a self-encoder, and then normal values that will not be identified by the anomaly detection system are generated based on the current abnormal values of the sensor and the actuator, but the method needs to monitor the industrial control system for a long time to obtain sufficient training data, and the existing method has poor attack effect in the limited covert attack situation.

[0009] In summary, although the existing covert attack method can realize covert attack in the black-box scenario and the white-box scenario, the attack effect of the existing covert attack method in the limited scenario in the black-box scenario is poor, and it is difficult to effectively evaluate the resistance of the anomaly detection system to the covert attack.

[0010] It should be noted that: the background art is only used to introduce the related information of the present application, so as to help understand the technical scheme of the present application, but does not mean that the related information must be prior art. In the absence of evidence that the related information has been disclosed before the filing date of the present application, the related information should not be regarded as prior art. SUMMARY

[0011] Therefore, the purpose of the present application is to overcome the defects of the prior art, and to provide a covert attack system based on reinforcement learning applied to an industrial control system and an anomaly detection evaluation method.

[0012] The object of the present application is achieved by the following technical solutions. According to a first aspect of the present application, a reinforcement learning-based covert attack system applied to an industrial control system is provided for performing a covert attack on the industrial control system, the industrial control system comprising a physical layer and a monitoring and control layer, wherein the physical layer comprises a plurality of sensors and a plurality of actuators, each actuator being configured to execute an operation instruction and record operation data, and each sensor being configured to monitor environmental data of the industrial control system; the monitoring and control layer comprises a data acquisition and monitoring control system and a plurality of programmable logic controllers, each programmable logic controller being configured to receive the environmental data sent by a corresponding sensor and the operation data sent by a corresponding actuator, and adjust the behavior of the corresponding actuator based on a control logic pre-configured by itself; the data acquisition and monitoring control system is configured to receive the environmental data and the operation data sent by all programmable logic controllers; the covert attack system comprises: a data acquisition module configured to intercept the environmental data and the operation data of each programmable logic controller sent to the data acquisition and monitoring control system in a preset deception manner when the industrial control system is attacked; an agent configured to add perturbations to the environmental data and the operation data of each current time window to obtain the environmental data and the operation data of each tampered current time window; wherein the agent is configured to use the environmental data and the operation data of the current time window, the environmental data and the operation data of the historical time window, and the known attackable physical layer data as state inputs, the environmental data and the operation data of the tampered current time window as action outputs, and use a preset reinforcement learning method to perform a plurality of rounds of iterative learning to obtain a deep neural network model; and a data transmission module configured to send the environmental data and the operation data of all tampered current time windows to the data acquisition and monitoring control system.

[0013] In some embodiments of the present application, the preset deception manner comprises a MAC address deception manner, a DHCP address deception manner, and an ARP deception manner.

[0014] In some embodiments of the present application, the agent is configured to use the environmental data and the operation data of the current time window, the environmental data and the operation data of the historical time window, and the known attackable physical layer data as state inputs, the environmental data and the operation data of the tampered current time window as action outputs, and use a preset reinforcement learning method to perform a plurality of rounds of iterative learning to obtain a deep neural network model, wherein a plurality of steps of iterative updating are performed in each round of iterative learning.

[0015] In some embodiments of the present application, the industrial control system further comprises a detection layer configured to receive the environment data and the operation data sent by the data acquisition and monitoring control system and perform anomaly detection, and in each round of iterative learning, each step of the iterative updating process is: obtaining the environment data in the current time window to be sent to the corresponding programmable logic controller by each sensor in the physical layer and the operation data in the current time window to be sent to the corresponding programmable logic controller by each actuator in the physical layer; taking all the environment data and operation data in the current time window, the environment data and operation data in the historical time window, and the known attackable physical layer data as state inputs to obtain the tampered environment data and operation data in each current time window; sending the tampered environment data and operation data in each current time window to the corresponding programmable logic controller, so that each programmable logic controller sends the corresponding tampered environment data and operation data in the current time window to the data acquisition and monitoring control system, and waits for the data acquisition and monitoring control system to send the tampered environment data and operation data in all current time windows to the detection layer for anomaly detection to obtain a detection result; and updating the parameters of the deep neural network model constructed based on the reinforcement learning method based on all the environment data and operation data in the current time window, the environment data and operation data in the historical time window, the known attackable physical layer data, the tampered environment data and operation data in all current time windows, and the detection result, using a preset reinforcement learning method.

[0016] In some embodiments of the present application, the preset reinforcement learning method is a deep deterministic policy gradient method, an advantage actor-critic method, an asynchronous advantage actor-critic method, a trust region policy optimization method, or a proximal policy optimization method.

[0017] According to a second aspect of the present application, an anomaly detection evaluation method is provided for evaluating the anomaly detection capability of an industrial control system when facing a covert attack, the method comprising: step S1, performing a covert attack on the industrial control system using the system according to the first aspect of the present application; and step S2, obtaining a detection result obtained by the detection layer in the industrial control system, wherein when the detection result is no anomaly, the anomaly detection fails; otherwise, the anomaly detection succeeds.

[0018] In some embodiments of the present application, the detection layer is configured with an anomaly detector configured to detect whether the environment data and the operation data sent by the data acquisition and monitoring control system are abnormal.

[0019] In some embodiments of the present application, the anomaly detector is configured to perform anomaly detection on the environment data and the operation data based on a residual-based method or an invariant-based method.

[0020] Compared with the prior art, the advantages of the present application are that: (1) an agent based on reinforcement learning is introduced, which directly interacts with the industrial control system through reinforcement learning to learn, which can save the intermediate link of operation data collection and improve the learning efficiency; (2) Furthermore, the agent can learn the operation logic and anomaly detection logic of the industrial control system at the same time, compared with the existing separate learning of the operation logic of the industrial control system or separate learning of the anomaly detection logic, the learning of the agent is more sufficient, which can improve the success rate of covert attack; (3) The input of the agent includes known attackable physical layer data set as a mask vector, which can improve the richness of the learning experience of the agent, and thus make the agent suitable for different restricted scenarios; (4) The reinforcement learning-based covert attack system can be applied to any process data-based anomaly detector, which widens the use range of covert attack. BRIEF DESCRIPTION OF DRAWINGS

[0021] The embodiments of the present application will be further described below with reference to the accompanying drawings, in which:

[0022] Figure 1 It is a schematic diagram of the covert attack system according to the embodiment of the present application;

[0023] Figure 2 It is a schematic diagram of the agent iterative learning process according to the embodiment of the present application;

[0024] Figure 3 It is a schematic diagram of the anomaly detection evaluation example according to the embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0026] As mentioned in the background section, although the existing covert attack method can achieve covert attack in black box and white box scenarios, the existing covert attack method has poor attack effect in restricted scenarios in black box scenario, and it is difficult to effectively evaluate the resistance ability of the anomaly detection system when facing covert attack.

[0027] To solve the above problems, the inventors studied the existing learning-based concealment attack method and found that the learning-based concealment attack method needs to monitor and obtain the operation data of the industrial control system for a long time, which leads to two problems: one is that long-time data collection is increasingly difficult, and the other is that it is difficult to determine the appropriate amount of data, and insufficient data reduces the learning effect and thus reduces the efficiency of the concealment attack. Based on the analysis of the existing learning-based concealment attack method, the inventors propose a new concealment attack system, in which an agent is set to tamper with the data of the sensors and actuators of the physical layer of the industrial control system. The agent learns directly with the industrial control system through reinforcement learning, which can save the intermediate link of operation data collection and improve the learning efficiency. Moreover, the agent can learn the operation logic and anomaly detection logic of the industrial control system at the same time, which is more sufficient than the existing separate learning of the operation logic or the separate learning of the anomaly detection logic of the industrial control system, and can improve the success rate of the concealment attack.

[0028] In summary, as shown in Figure 1 The present application provides a reinforcement learning-based concealment attack system applied to an industrial control system, which is used for performing a concealment attack on the industrial control system. The industrial control system includes a physical layer and a monitoring and control layer. The physical layer includes a plurality of sensors and a plurality of actuators. Each actuator is used to execute an operation instruction and record operation data. Each sensor is used to monitor environmental data of the industrial control system. The monitoring and control layer includes a data acquisition and monitoring control system and a plurality of programmable logic controllers. Each programmable logic controller is used to receive environmental data sent by a corresponding sensor and operation data sent by a corresponding actuator, and adjust the behavior of the corresponding actuator based on a control logic pre-configured by itself. The data acquisition and monitoring control system is used to receive environmental data and operation data sent by all programmable logic controllers. The concealment attack system includes a data acquisition module, an agent, and a data transmission module. The data acquisition module is used to intercept environmental data and operation data of a current time window sent by each programmable logic controller to the data acquisition and monitoring control system in a preset deception manner when the industrial control system is attacked. The agent is used to add disturbance to the environmental data and operation data of each current time window to obtain tampered environmental data and operation data of each current time window. The agent is a deep neural network model obtained by performing multiple rounds of iterative learning using a preset reinforcement learning method, with the environmental data and operation data of the current time window, the environmental data and operation data of a historical time window, and known attackable physical layer data as state inputs, and the tampered environmental data and operation data of the current time window as action outputs. The data transmission module is used to send all tampered environmental data and operation data of the current time window to the data acquisition and monitoring control system.

[0029] Before the embodiments of the present application are described in detail, the hidden attack mode for the industrial control system is briefly introduced.

[0030] The hidden attack refers to that an attacker intercepts the physical layer data transmitted by the physical layer of the industrial control system by using technical means, adds disturbance to the physical layer data and returns the physical layer data with disturbance to the industrial control system, so that the industrial control system cannot detect the anomaly in the physical layer data with disturbance, thereby achieving the purpose of hidden attack.

[0031] Specifically, the process of the hidden attack can be formalized as the following process.

[0032] The anomaly detection of the industrial control system on the physical layer data (data of sensors and actuators) transmitted by the physical layer should meet: wherein, represents an anomaly detection function, represents that there is an anomaly in the underlying physical layer; the purpose of the hidden attack is to seek , so that , that is, disturbance is added to the physical layer data to obtain , so that the industrial control system cannot detect the anomaly that is occurring.

[0033] In order to better understand the present application, the following will describe each component of the hidden attack system in detail in combination with specific embodiments.

[0034] I. Data acquisition module

[0035] The data acquisition module is used to intercept the environmental data and operation data of the current time window sent by each programmable logic controller to the data acquisition and monitoring control system in a preset deception mode when the industrial control system is attacked.

[0036] According to an embodiment of the present application, the preset deception mode includes a MAC address deception mode, a DHCP address deception mode and an ARP deception mode.

[0037] It should be noted that the industrial control system can be an actual industrial control system in the real world, or can be a simulated industrial control system constructed according to the type and configuration information of the industrial control system by using simulation software or a simulation system. For example, an open-source platform such as a water treatment simulation platform DHALSIM provided by a research institution, or a commercial simulation software such as a Simcenter series simulation software provided by a manufacturer. It should be further noted that if the stealth attack is launched by an enemy, the configuration information of the industrial control system can come from an attacker lurking in the industrial control system, or intelligence leakage caused by insecure communication of the owner of the industrial control system; if the stealth attack is initiated by the owner of the industrial control system to evaluate the resistance of the current anomaly detection system to the stealth attack, the configuration information of the industrial control system can be directly provided by the industrial control system.

[0038] II. Agent

[0039] The agent is configured to add perturbations to the environment data and operation data of each current time window to obtain tampered environment data and operation data of each current time window; wherein the agent is configured to take the environment data and operation data of the current time window, the environment data and operation data of the historical time window, and the known attackable physical layer data as state input, take the tampered environment data and operation data of the current time window as action output, and take a deep neural network model obtained by performing multiple rounds of iterative learning by using a preset reinforcement learning method.

[0040] According to an embodiment of the present application, the agent is configured to take the environment data and operation data of the current time window, the environment data and operation data of the historical time window, and the known attackable physical layer data as state input, take the tampered environment data and operation data of the current time window as action output, and take a deep neural network model obtained by performing multiple rounds of iterative learning by using a preset reinforcement learning method, wherein multiple-step iterative updates are performed in each round of iterative learning.

[0041] According to one embodiment of the present invention, the industrial control system further includes a detection layer for receiving environmental data and operation data sent by the data acquisition and monitoring control system and performing anomaly detection. In each round of iterative learning, each iterative update process is as follows: obtaining the environmental data of the current time window that each sensor in the physical layer wants to send to the corresponding programmable logic controller, and the operation data of the current time window that each actuator in the physical layer wants to send to the corresponding programmable logic controller; using the obtained environmental data and operation data of all current time windows, the environmental data and operation data of historical time windows, and known attackable physical layer data as state inputs to obtain the environmental data and operation data of each current time window after tampering; using the tampered environmental data and operation data of each current time window after tampering The environmental data and operation data of the previous time window are sent to the corresponding programmable logic controller, so that each programmable logic controller sends the corresponding tampered environmental data and operation data of the current time window to the data acquisition and monitoring control system, and waits for the data acquisition and monitoring control system to send the tampered environmental data and operation data of all current time windows to the detection layer for anomaly detection to obtain the detection results; based on the environmental data and operation data of all current time windows, the environmental data and operation data of historical time windows, the known attackable physical layer data, the tampered environmental data and operation data of all current time windows and the detection results, the preset reinforcement learning method is used to update the parameters of the deep neural network model constructed based on the reinforcement learning method.

[0042] According to one embodiment of the present invention, the preset reinforcement learning is: a deep deterministic policy gradient method, a dominant actor-critic method, an asynchronous dominant actor-critic method, a trust region policy optimization method, or a proximal policy optimization method.

[0043] To gain a deeper understanding of the iterative learning process of intelligent agents, we briefly introduce the deep deterministic policy gradient method as an example. Other reinforcement learning methods, like the deep deterministic policy gradient method, are relatively mature reinforcement learning technologies and will not be discussed in detail here.

[0044] like Figure 2 As shown, the intelligent agent interacts with the industrial control system and acquires a deep neural network model through multiple rounds of iterative learning based on a pre-defined reinforcement learning method. Through these multiple rounds of iterative learning, the intelligent agent fully learns the operational logic of the industrial control system and the anomaly detection logic of the detection layer. It then adds appropriate perturbations based on abnormal sensor and actuator values ​​to evade detection at the detection layer. Each round of iterative learning involves multiple iterative updates, each of which proceeds as follows.

[0045] First, the current time window environment data of each sensor in the physical layer of the industrial control system to be sent to the corresponding programmable logic controller and the current time window operation data of each actuator in the physical layer to be sent to the corresponding programmable logic controller are obtained, which represent the running state of the underlying physical layer, and these data may be normal or abnormal, and the obtained data is represented as a numerical matrix , wherein represents the current time window environment data and operation data, represents the historical time window environment data and operation data, and the numerical matrix records the data of time windows. It should be noted that the historical time window environment data and operation data are obtained from the physical layer of the industrial control system in the previous time window. It should be noted that, since a large number of interactions are required for training, the real-world industrial control system cannot bear the cost of training, therefore, in the process of iterative learning, the agent interacts with the simulated industrial control system to learn the running logic of the industrial control system and the anomaly detection logic of the detection layer.

[0046] Then, the attackable physical layer data, i.e., the sensors and actuators that can be tampered with by the attacker, is determined, which is represented as a binary mask vector . The mask vector is a method of selectively operating or screening corresponding bits in the original vector by setting bits in the binary vector. Specifically, each bit in the mask vector corresponds to a sensor or an actuator, when a bit in the mask vector is 1, it means that the sensor or actuator corresponding to the bit can be tampered with; when a bit in the mask vector is 0, it means that the sensor or actuator corresponding to the bit cannot be tampered with. It should be noted that the mask vector can be adjusted according to the attack or iterative learning requirements.

[0047] Then, the numerical matrix and the mask vector form the state input of the agent, and the agent makes a decision action based on the state input , i.e., adds disturbance to the current time window environment data and operation data to obtain tampered data .

[0048] Then, the tampered data send to the corresponding programmable logic controller (PLC) to make each programmable logic controller send the corresponding tampered current time window environment data and operation data to the data acquisition and monitoring control system (SCADA), wait for the data acquisition and monitoring control system to send all tampered current time window environment data and operation data to the detection layer for anomaly detection to obtain the detection result; at the same time, each programmable logic controller adjusts the behavior of the actuator based on the pre-configured control logic after receiving the tampered data, thereby changing the physical layer state of the industrial control system to obtain the state input of the next iteration learning It should be noted that the control logic is written in the programmable logic controller in the form of code to control the actuator, for example, the control logic is to close the valve when the environment temperature is greater than or equal to 80 degrees, and for example, the control logic is to open the motor when the environment humidity is greater than or equal to the preset threshold.

[0049] Finally, based on all current time window environment data and operation data, historical time window environment data and operation data, known attackable physical layer data, all tampered current time window environment data and operation data, and detection results, the parameters of the deep neural network model constructed based on the reinforcement learning method are updated using a preset reinforcement learning method. It should be noted that if the anomaly detection result is abnormal, or the maximum training step is reached, the current training is ended; if the anomaly detection result is normal (normal), the agent receives a certain reward (return), and the reward is set to a positive number to encourage the agent to perform more steps of iteration learning with the industrial control system, and the agent is updated according to the <state, action, reward, new state> according to the update policy of the deep deterministic policy gradient method, completing the current iteration learning. It should also be noted that since the attacker cannot obtain the anomaly detection principle in the industrial control system in the black box scenario, the detection layer in the simulated industrial control system can be set to a combination of residual-based and invariant-based anomaly detection methods when constructing the agent, which can improve the robustness of the constructed agent and also broaden its application scope.

[0050] It should be noted that through the iteration learning process as shown in Figure 2 The agent can learn the operation logic and anomaly detection logic of the industrial control system simultaneously through the iteration learning process, and compared with the existing separate learning of the operation logic or separate learning of the anomaly detection logic of the industrial control system, the learning of the agent is more sufficient, which can improve the success rate of covert attacks.

[0051] III. Data transmission module

[0052] The data transmission module is used to send all tampered current time window environment data and operation data to the data acquisition and monitoring control system.

[0053] Based on the stealthy attack system described in the foregoing embodiments, the present application further proposes an anomaly detection evaluation method for evaluating the anomaly detection capability of an industrial control system when facing a stealthy attack, which comprises the following steps: step S1, performing a stealthy attack on the industrial control system by using the stealthy attack system described in the foregoing embodiments; and step S2, obtaining the detection result obtained by the detection layer in the industrial control system, and when the detection result is no anomaly, the anomaly detection fails; otherwise, the anomaly detection succeeds. It should be noted that in the anomaly detection evaluation process, the anomaly detection capability of a real-world industrial control system when facing a stealthy attack is evaluated.

[0054] According to an embodiment of the present application, the detection layer is configured with an anomaly detector for detecting whether the environment data and operation data sent by the data acquisition and monitoring control system are abnormal.

[0055] According to an embodiment of the present application, the anomaly detector is configured to perform anomaly detection on the environment data and operation data in a residual-based manner or an invariant-based manner. In the residual-based manner, a detection model is trained by minimizing the loss function (generally using the mean square error) between the expected value and the observed value or between the input and the output, and an abnormal alarm will be triggered when the difference exceeds a set threshold; this anomaly detection method includes the autoregressive (AR), autoencoder (AE) and other deep neural network (DNN) models. In the invariant-based manner, a set of rules that are always met when the industrial control system is running normally are constructed to identify anomalies, and the construction of the rules often needs to be based on the detailed knowledge of a specific industrial control system, and once a rule is not met, it means that an anomaly has occurred; this detection method includes the SFIG (Systematic Framework for Invariant Generation). It should be noted that the anomaly detection method configured by the anomaly detector in the real-world industrial control system is flexibly configured according to the actual requirements, and the present application does not make special limitations.

[0056] In order to better understand the execution process of the anomaly detection evaluation method, the anomaly detection evaluation process shown in FIG. 1 is taken as an example to be described below. As shown in FIG. 1, the anomaly detection evaluation process comprises the following steps: Figure 3 Figure 3 ​It can be known that the preset cheating mode is used to intercept the environmental data and operation data of each programmable logic controller in the current time window sent to the data acquisition and monitoring control system; then, the agent is used to tamper with the intercepted environmental data and operation data in the current time window; the tampered environmental data and operation data in the current time window are sent to the data acquisition and monitoring control system, so that the data acquisition and monitoring control system sends all the tampered environmental data and operation data in the current time window to the detection layer for abnormality detection; finally, the detection result is obtained, and when the detection result is normal (no exception), it indicates that the abnormality detection fails.

[0057] The beneficial effects of the present application are: (1) the agent based on reinforcement learning is introduced, which directly interacts with the industrial control system through the way of reinforcement learning, can save the intermediate link of operation data collection, and improve the learning efficiency; (2) the agent can learn the operation logic and abnormality detection logic of the industrial control system at the same time, compared with the existing separate learning of the operation logic or separate learning of the abnormality detection logic of the industrial control system, the learning of the agent is more sufficient, and the success rate of the covert attack can be improved; (3) the input of the agent includes the known attackable physical layer data set as a mask vector, which can improve the richness of the learning experience of the agent, and then make the agent suitable for different restricted scenes; (4) the covert attack system based on reinforcement learning can be applied to any process data based abnormality detector, which widens the use range of the covert attack.

[0058] It should be noted that although the above describes each step in a specific order, it does not mean that each step must be executed in the above specific order, in fact, some of these steps can be executed concurrently, or even the order is changed, as long as the required function can be realized.

[0059] The present application can be a system, a method and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions loaded thereon for causing a processor to implement various aspects of the present application.

[0060] A computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves.

[0061] Embodiments of the application have been described above, with the understanding that these embodiments are exemplary only, and are not restrictive, in terms of the scope of the embodiments disclosed. Many modifications and variations of the described embodiments are possible, in light of the above teachings, without departing from the scope and spirit of the described embodiments. The choice of words in this document is intended to best explain the principles of the embodiments, practical application, or technical improvement in the art, or to enable others skilled in the art to utilize the embodiments disclosed herein.

Claims

1. A covert attack system based on reinforcement learning applied to an industrial control system, for performing covert attacks on the industrial control system, wherein the industrial control system includes a physical layer and a monitoring and control layer, wherein: The physical layer includes multiple sensors and multiple actuators. Each actuator is used to execute operating instructions and record operating data. Each sensor is used to monitor the environmental data of the industrial control system. The monitoring and control layer includes a data acquisition and monitoring control system and multiple programmable logic controllers. Each programmable logic controller is used to receive environmental data sent by the corresponding sensor and operating data sent by the corresponding actuator, and adjust the behavior of the corresponding actuator based on its own pre-configured control logic. The data acquisition and monitoring control system is used to receive environmental data and operation data sent by all programmable logic controllers; it is characterized in that the covert attack system includes: A data acquisition module is used to intercept the environmental data and operation data of the current time window sent by each programmable logic controller to the data acquisition and monitoring control system in a preset deceptive manner when the industrial control system is attacked; An intelligent agent, configured to add perturbations to the environmental data and operational data of each current time window to obtain the tampered environmental data and operational data of each current time window; wherein the intelligent agent uses the environmental data and operational data of the current time window, the environmental data and operational data of the historical time window, and known attackable physical layer data as state inputs, and the tampered environmental data and operational data of the current time window as action outputs, and uses a preset reinforcement learning method to execute multiple rounds of iterative learning to obtain a deep neural network model; The data transmission module is used to send all the tampered environmental data and operation data of the current time window to the data acquisition and monitoring control system.

2. The system according to claim 1, wherein: The preset spoofing methods include: MAC address spoofing method, DHCP address spoofing method and ARP spoofing method.

3. The system according to claim 2, characterized in that The intelligent agent uses the environmental data and operation data of the current time window, the environmental data and operation data of the historical time window, and the known attackable physical layer data as state input, and the tampered environmental data and operation data of the current time window as action output. It uses a preset reinforcement learning method to perform multiple rounds of iterative learning to obtain a deep neural network model, wherein multiple steps of iterative updates are performed in each round of iterative learning.

4. The system according to claim 3, characterized in that The industrial control system also includes a detection layer, which is used to receive environmental data and operation data sent by the data acquisition and monitoring control system and perform anomaly detection. In each round of iterative learning, the iterative update process is as follows: Obtaining environmental data of a current time window that each sensor in the physical layer intends to send to the corresponding programmable logic controller, and operation data of a current time window that each actuator in the physical layer intends to send to the corresponding programmable logic controller; Use all acquired environmental data and operation data of the current time window, environmental data and operation data of the historical time window, and known attackable physical layer data as state input to obtain the tampered environmental data and operation data of each current time window; Sending the tampered environmental data and operation data of each current time window to the corresponding programmable logic controller, so that each programmable logic controller sends the corresponding tampered environmental data and operation data of the current time window to the data acquisition and monitoring control system, waiting for the data acquisition and monitoring control system to send the tampered environmental data and operation data of all current time windows to the detection layer for anomaly detection to obtain the detection results; Based on the environmental data and operation data of all current time windows, the environmental data and operation data of historical time windows, the known attackable physical layer data, the tampered environmental data and operation data of all current time windows and the detection results, a preset reinforcement learning method is used to update the parameters of the deep neural network model constructed based on the reinforcement learning method.

5. The system according to claim 4, characterized in that The default reinforcement learning methods are: deep deterministic policy gradient method, advantage actor-critic method, asynchronous advantage actor-critic method, trust region policy optimization method or proximal policy optimization method.

6. An anomaly detection evaluation method for evaluating the anomaly detection capability of an industrial control system when facing covert attacks, characterized in that: The method comprises: Step S1: Using the system according to any one of claims 1 to 5 to perform a covert attack on an industrial control system; Step S2: Obtain the detection result obtained by the detection layer in the industrial control system. When the detection result is no abnormality, the abnormality detection fails; otherwise, the abnormality detection succeeds.

7. The method according to claim 6, characterized in that The detection layer is equipped with an anomaly detector for detecting whether there are anomalies in the environmental data and operation data sent by the data acquisition and monitoring control system.

8. The method according to claim 7, characterized in that The anomaly detector is configured to perform anomaly detection on the environmental data and the operation data in a residual-based manner or an invariant-based manner.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any one of the methods of claims 6-8.

10. An electronic device, characterized in that: include: one or more processors; and a memory, wherein the memory is configured to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 6 to 8 by executing the executable instructions.

Images