Industrial Internet of Things honeypot deployment method and device based on adversarial reinforcement learning

Abstract

The invention discloses an industrial Internet of Things honeypot deployment method and device based on adversarial reinforcement learning, and belongs to the technical field of network security. The problems of dynamics and uncertainty of network attack behaviors are mainly solved. According to the implementation scheme, the method comprises the following steps: collecting industrial Internet of Things equipment data including an industrial Internet of Things equipment IP, industrial Internet of Things equipment request data and industrial Internet of Things equipment response data; carrying out clustering analysis on the request data by utilizing an LDA analysis method based on the industrial Internet of Things multi-mode request data, and dividing the request data into different categories; constructing an interaction model of the honeypot and an attacker by using a Markov decision model; and finally, the model is trained by using a Repeated Update Q-learning (RUQL) algorithm. According to the method, network attacks can be effectively attracted and coped, and the capability of capturing and analyzing the network attacks in an adversarial environment is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of network security, and in particular relates to an industrial Internet of Things honeypot deployment method and device based on adversarial reinforcement learning. Background Art

[0002] In recent years, with the popularization of industrial Internet of Things technology, its security issues have become increasingly prominent and have become the focus of academic and industrial circles. Industrial Internet of Things devices and systems are widely used in fields closely related to personal safety, such as smart medical care, security, transportation or manufacturing. Its security issues are not only related to the safety of human life and property, but also to social stability and national security.

[0003] Compared with passive protection methods such as intrusion detection and firewalls, honeypots, as a network security protection method that actively understands the attack situation, aim to attract and monitor the activities of attackers by deploying a fake system disguised as a real system. A successful honeypot can simulate various services of a real system, including operating systems, applications, databases, etc., to provide an interactive experience close to that of a real device. Once an attacker begins to conduct activities on the honeypot, the honeypot will record their behavior data. This data includes the attacker's IP address, attack time, attack method, tools used, etc. By analyzing the attack data collected by the honeypot, the security team can understand the type, source and trend of current threats. Therefore, in order to better study and analyze new and unknown industrial IoT attacks, industrial IoT honeypot technology has become an important research field in IoT security.

[0004] In recent years, there have been innovative studies that incorporate reinforcement learning concepts into honeypot technology. Intelligent high-interaction honeypots based on general reinforcement learning use algorithms to dynamically adjust and optimize the security mechanism of honeypot behavior. It learns and predicts the attacker's next move by analyzing the attacker's behavior pattern. However, in previous intelligent honeypots based on general reinforcement learning, the attackers in a steady state were studied, that is, it was believed that the state transitions of all attackers did not change over time, and the feedback on the same honeypot action was homogenized. But in real physical confrontation scenarios, the attacker's feedback on the same honeypot action may be different. This means that even when facing the same honeypot action, the same attacker may give different feedback at different times. Therefore, intelligent honeypots based on general reinforcement learning do not take into account the dynamics and uncertainty of network attack behavior and cannot cope with complex attack scenarios. Summary of the invention

[0005] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides an industrial Internet of Things honeypot deployment method and device based on adversarial reinforcement learning, which solves the problem that it is difficult to effectively capture and analyze network attacks under non-stationary conditions.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] According to a first aspect of this specification, a method for deploying an industrial Internet of Things honeypot based on adversarial reinforcement learning is provided, the method comprising the following steps:

[0008] (1) Collect IIoT device data, including IIoT device IP, IIoT device request data, and IIoT device response data;

[0009] (2) Based on the IIoT device request data collected in step (1), the LDA analysis method based on IIoT multimodal request data is used to perform data clustering and classify the request data into different categories;

[0010] (3) Based on the classified request data in step (2), a Markov decision model is used to construct an interaction model between the honeypot and the attacker, where the honeypot is the intelligent agent, the network attacker attacking the honeypot is the environment, the state is the classified network attacker's request, the action is the response data replied by the honeypot, and the request to induce attack and extend the session is rewarded;

[0011] (4) Based on the Markov decision model constructed in step (3), the adversarial reinforcement learning algorithm RUQL is used to train the model. The honeypot selects the optimal response to respond to attackers with complex and changeable behaviors.

[0012] Furthermore, the collecting of industrial Internet of Things device data specifically includes: collecting real industrial Internet of Things device request data and IP; traversing the IP to open the TCP socket, actively sending TCP requests to the industrial Internet of Things devices, observing and collecting their response data and storing them in the response database.

[0013] Furthermore, in step (1), the method of collecting request data is: deploying a low-interaction honeypot instance to monitor the bait port to collect request data, or using the penetration tool Burpsuit to collect request data.

[0014] Furthermore, step (2) is specifically as follows: each industrial Internet of Things device request is regarded as a document, and the predefined request data category is regarded as the topic of the document; each document is divided into a series of words by a predefined separator; the words of all documents are combined together to form a global corpus; the frequency of occurrence of each word in the corpus is calculated; the number of predefined request data categories is used as the number of topics, and the input request data category can be obtained after training the LDA model.

[0015] Furthermore, the categories of the requested data include: attack data targeting industrial Internet of Things devices, attack data containing Linux Bash instructions, attack data of malware upload type, other attack data, and termination status data.

[0016] Furthermore, step (2) also includes a data preprocessing step, specifically: removing invalid or erroneous data records, and standardizing the request data so that the mean of each feature is 0 and the standard deviation is 1.

[0017] Furthermore, in step (3), the goal of state definition is to capture the key features of the interaction between the attacker and the honeypot while keeping the state space relatively compact; the states of the interaction between the attacker and the honeypot are divided into five state sets, namely S = {s1, s2, s3, s4, s5}; s1 is the attack data against industrial IoT devices, including vulnerability exploitation, permission bypass, brute force cracking, malicious scanning, and access attempts; s2 is the attack data containing Linux Bash instructions; s3 is the attack data of malware upload type, and this type of attack is identified by interpreting the file suffix; s4 is other attack data, which refers to requests that cannot be processed, including TLS communications composed of hexadecimal data; s5 is the termination state data, and when the agent does not receive the next state or times out, it will automatically enter the termination state.

[0018] Furthermore, in step (4), the RUQL algorithm considers the difference between the actual return and the predicted Q value and adjusts the update according to the difference between the strategy and the behavior. The Q value update formula is:

[0019]

[0020] Among them, the expression Measures the difference between the current Q-value prediction and the actual Q-value, r is the reward obtained after performing action a in state s; γ is the discount factor; is the maximum Q value of all possible actions in the next state s′; α is the learning rate; π(s,a) is the probability that the strategy chooses action a in state s, is a weight used to adjust the update to reflect the difference between the policy and the behavior; β is a tuning parameter used to prevent the policy from approaching zero.

[0021] Furthermore, in step (4), the specific operation steps of the RUQL algorithm are: a. Observe the current state s and arbitrarily initialize the value function Q; b. For each time step t, the agent selects action a according to the current value function Q and exploration strategy, outputs action a and observes a result reward r and the next state s′ of the environment; updates the value function Q using the Q value update formula in RUQL; repeats step b until the value function Q converges.

[0022] According to a first aspect of the present specification, there is provided an industrial Internet of Things honeypot deployment device based on adversarial reinforcement learning, comprising three components, which operate independently and share data with each other during the learning process; specifically comprising:

[0023] A response database for storing the obtained response data about the industrial Internet of Things devices;

[0024] The data receiving module is used to capture attack requests and simplify the requests using the LDA algorithm;

[0025] The adversarial interaction module uses the RUQL algorithm to train the model based on the given response feedback of the attacker. When dealing with attackers with complex and changeable behaviors, the honeypot continuously learns and adapts to the changes in network attacks, and dynamically adjusts the behavior and strategy of the honeypot based on the observed attack behavior to more effectively attract and respond to attacks.

[0026] In the attack traffic classification research, when collecting a large amount of request data from real industrial IoT devices, it was found that the original request data packet may lead to a sparse state space. Therefore, after investigating a large amount of literature, a machine learning method of generative text clustering was used to perform text clustering on the original request data packet. Through in-depth analysis of the attack traffic classification research, the present invention realizes an industrial IoT honeypot system based on adversarial reinforcement learning, which solves the problem of difficulty in effectively capturing and analyzing network attacks under non-stationary conditions. At the same time, with the goal of increasing the amount of intelligence collected in the industrial IoT honeypot interaction process and the depth of attack interaction, an adversarial reinforcement learning method was selected to realize the construction of the honeypot system, and the effectiveness of the method was verified. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 It is a flow chart of an industrial Internet of Things honeypot deployment method based on adversarial reinforcement learning provided by an embodiment of the present invention;

[0029] Figure 2 is a schematic diagram of cluster analysis of LDA request data provided by an embodiment of the present invention;

[0030] Figure 3 The present invention provides an embodiment of the present invention for constructing an MDP model in a real adversarial scenario. DETAILED DESCRIPTION

[0031] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Reference Figure 1 The embodiment of the present invention provides an industrial Internet of Things honeypot deployment method based on adversarial reinforcement learning, the method comprising the following steps:

[0033] (1) Data collection of IIoT devices; the ultimate goal of data collection is to collect response resources. Therefore, the first step in building a honeypot is to collect real IIoT device request data and IP, then traverse the IP to open the TCP socket, actively send TCP requests to IIoT devices, observe and collect their response data and store them in the response database.

[0034] The present invention can collect request data in the following ways:

[0035] 1. Deploy a low-interaction honeypot instance on the Tencent Cloud server to collect relevant request data;

[0036] 2. Use the penetration tool Burpsuit to collect relevant request data. This part of request data is of higher quality than the request data collected by the low-interaction honeypot.

[0037] The present invention uses the third-party platform Shodan to obtain the original IP information in IP collection. Its working principle is to crawl publicly accessible device information on the Internet, such as the IP address and metadata of cameras, printers and routers. This information may include the model, manufacturer, open port, service type, etc. of the device.

[0038] In response data collection, the present invention determines the IP address of the device to be accessed, and uses the Requests library of Python to write a program to send requests and receive responses.

[0039] (2) Based on the IIoT device request data collected in step (1), data clustering is performed using the LDA (Latent Dirichlet Allocation) analysis method based on the IIoT multimodal request data. The IIoT device request data contains multidimensional features, including but not limited to attack methods, timestamps, device status, operation types, etc. These features can more comprehensively reflect the operating status of the device and the request content.

[0040] In the data preprocessing stage, invalid or erroneous data records are first removed, for example, records with missing key information are deleted, and obviously erroneous data are corrected.

[0041] Secondly, in order to eliminate the dimensional impact between request data from different devices, the request data is standardized so that the mean of each feature is 0 and the standard deviation is 1.

[0042] Next, each industrial Internet of Things device request is regarded as a document, and the predefined request data category is regarded as the subject of the document; the request data categories include: attack data against industrial Internet of Things devices, attack data containing Linux Bash instructions, attack data of malware upload type, other attack data, and termination status data, and each of the above categories can also contain several subcategories; the present invention divides each document into a series of words through a predefined separator; then the words of all documents (requests) are combined together to form a global corpus; next, the occurrence frequency (statistical distribution) of each word in the corpus is calculated; the number of predefined request data categories is used as the number of topics n, and the input request data category can be obtained after training the LDA model.

[0043] The above method applies the LDA model to cluster analysis of IIoT device request data. Through multi-dimensional feature extraction and customized clustering algorithms, it achieves in-depth understanding and dynamic monitoring of device request data, and improves the accuracy and real-time performance of data analysis. Combining the LDA model in the field of natural language processing with the field of IIoT realizes cross-domain technology integration, which provides a new perspective and method for IIoT data analysis.

[0044] (3) Based on the classified request data in step (2), the present invention uses a Markov decision model to construct an interaction model between the honeypot and the attacker, where the honeypot is the intelligent agent; the network attacker attacking the honeypot is the environment; the state is the classified network attacker's request. More specifically, the state represents the current situation of the attacker interacting with the honeypot.

[0045] The goal of the state definition in the present invention is to capture the key features of the interaction between the attacker and the honeypot while keeping the state space relatively compact. Using the network kill chain model, the present invention divides the interaction process between the attacker and the honeypot into seven stages, namely, reconnaissance, weaponization, delivery, exploitation, installation, command control, and goal achievement. For example, the classification of requests based on HTTP (such as GET, POST, PUT, etc.) or higher-level protocol types (such as FTP, SSH, etc.) can illustrate the malicious scanning behavior of the attacker in the reconnaissance stage. In addition, the classification based on the request target path (using the target URL path of the request as part of the state) can provide information about the weaponization stage. The goal of the embodiment of the present invention is to capture the key features of the interaction between the attacker and the honeypot while keeping the state space relatively compact. Therefore, the embodiment of the present invention divides the state of the interaction between the attacker and the honeypot into the following 5 state sets, a total of 24 states, namely S = {s1, s2, s3, s4, s5}.

[0046] s1: Attack data against industrial IoT devices, such as vulnerability exploitation, permission bypass, brute force cracking, malicious scanning, access attempts, etc. The specific content is shown in Table 1. s1 contains 15 states in total.

[0047] Table 1 Attack data against industrial IoT devices

[0048]

[0049]

[0050] s2: attack data containing Linux Bash instructions. According to the threat level, the embodiment of the present invention divides it into 4 states, and the specific contents are shown in Table 2. From bottom to top, the threat level increases in sequence.

[0051] Table 2 Attack data containing Linux Bash instructions

[0052] Linux Bash Commands kill,wegt,sudo,mount,umount,passwd cat,rm,tar,chmod,chown cd,cp,mkdir,rmdir,touch ls,pwd,ps,df

[0053] s3: malware upload attack data. This type of attack is common in WEB attacks and is highly harmful. It usually has obvious features in the file suffix. The embodiment of the present invention identifies this type of attack by interpreting the file suffix. The specific content is shown in Table 3 and is divided into three states.

[0054] Table 3 Attack data of malware upload types

[0055] Suffix Type Variants asp asp,aspx,asa,asax,ascx,ashx,asmx,aSax,aScx,aShx,aSmx,cEr php php,ppHp3,pHp2,html,htm,phtml,pht,Html,Htm,pHtml jsp jsp,jspa,jspx,jsw,jsv,jspf,jtml,jSp,jSpx,jSpa,jSw,jSv,jSpf,jHtml

[0056] s4: Other attack data. Refers to some requests that cannot be processed, such as some TLS communications consisting of hexadecimal data, so it is treated as a separate request category.

[0057] s5: Termination state data. When the agent does not receive the next state or times out, it will automatically enter the termination state.

[0058] (4) Based on the Markov decision model constructed in step (3), the adversarial reinforcement learning algorithm RUQL is used to train the model. The main problem solved by the RUQL algorithm is the policy deviation of action value update. The policy deviation problem occurs in the Q-learning (QL) algorithm because the value Q of an action is only updated when the action is executed. Therefore, the update rate of the action value directly depends on the probability of selecting the action for execution. Obviously, it is unrealistic to update the value Q of all actions at the same time in a single state, but the value Q of an action can be repeatedly updated in each state, so that the policy deviation problem is alleviated. Among them, the Q value update formula in RUQL is:

[0059]

[0060] The expressions in brackets are It measures the difference between the current Q-value prediction and the actual Q-value. r is the reward obtained after performing action a in state s. γ is the discount factor, which determines the current value of future rewards. is the maximum Q value of all possible actions in the next state s′. α is the learning rate, which controls the step size of the Q value update. π(s,a) is the probability that the policy chooses action a in state s, This weight is used to adjust the update to reflect the difference between the policy and the behavior; if the policy is more likely to choose action a, that is, π(s,a) is large, the weight is small; if the policy is less likely to choose action a, that is, π(s,a) is small, the weight is large. β is a tuning parameter that prevents the policy from approaching zero; when it approaches zero, the update rate becomes unbounded (approaching infinity), so in practice, safeguards must be added. The update formula of the RUQL algorithm updates the Q value by taking into account the difference between the actual reward and the predicted Q value, and adjusting the update according to the difference between the policy and the behavior.

[0061] The operation steps of the RUQL algorithm are as follows:

[0062] 1. Observe the current state s and arbitrarily initialize the value function Q.

[0063] 2. For each time step t, the agent chooses an action a based on the current value function Q and exploration strategy (such as ε-greedy).

[0064] 3. Output action a and observe a resulting reward r and the next state s of the environment ′ .

[0065] 4. Update the formula using the Q value in RUQL Update the value function Q. It is worth noting that This weight is used to adjust the update, which is inversely proportional to the probability of choosing the action, so that actions with a small probability of being chosen are updated more often.

[0066] 5. Repeat steps 2-4 until the value function Q converges.

[0067] The following examples will illustrate the advanced nature of the method of the present invention. After the honeypot system is deployed in an actual scenario, the performance of the adversarial interactive honeypot based on RUQL (the honeypot designed by the present invention, hereinafter referred to as RUQL honeypot), the honeypot based on general reinforcement learning QL (hereinafter referred to as QL honeypot) and the random honeypot are tested and compared.

[0068] A. Experimental Setup

[0069] The present invention has implemented an adversarial interactive honeypot based on RUQL and deployed it on the public Internet. Due to limited budget, an instance with 2 cores and 2G and a public network bandwidth of 4M is used in the configuration to run on the Tencent cloud server. In order to simulate the Hikvision camera, the present invention opens ports 80, 81, and 554. In order to verify the performance of the adversarial interactive honeypot based on RUQL, the present invention simultaneously deploys a honeypot based on general reinforcement learning QL (QL honeypot) and a random honeypot with the same configuration to run on the Tencent cloud server.

[0070] QL Honeypot: Deploy the general reinforcement learning algorithm QL.

[0071] Random honeypot: A honeypot where no algorithm is deployed and each response is selected with equal probability.

[0072] B. Evaluation Results

[0073] After the three honeypots were run on Tencent Cloud Server for nearly a month (April 8, 2024 - May 1, 2024), some basic data were collected, as shown in Table 4. The average session lengths of RUQL honeypot, QL honeypot and random honeypot were 4.9, 3.9 and 2.11 respectively. The number of attack sessions was 1908, 1159 and 425 respectively. Long sessions and multiple attack sessions indicate that the attacker attempts to deeply detect or exploit vulnerabilities, whereas it may be simple scanning or harmless traffic. The random honeypot simulates a decoy interface, with an average session length of about 2 times and fewer attack sessions. The RUQL honeypot and QL honeypot, two high-interaction honeypots, have long sessions and many attack sessions, providing rich services and in-depth detection of attacks. Since the QL honeypot attempts to learn a fixed optimal strategy, its evaluation indicators may be limited by the strategy it has learned. Since the RUQL honeypot can adapt to changes in the environment, its session length may be longer and the number of attack sessions may be greater. When the attacker’s behavior changes, this non-stationary reinforcement learning honeypot of RUQL honeypot can adjust its strategy to better simulate the real system and attract the attacker for deeper interaction.

[0074] Table 4 Basic data of different honeypots

[0075] RUQL Honeypot QL Honeypot Random Honeypot Average session length 4.9 3.9 2.11 Number of attack sessions 1908 1159 425

[0076] Table 5 shows the number and distribution of attack requests collected by different honeypots. The RUQL honeypot collects far more attack requests than the QL honeypot. This means that the non-stationary reinforcement learning method is more effective in honeypot design. The QL honeypot system under general reinforcement learning often attracts attackers based on static or fixed rule behavior patterns, while the RUQL honeypot system under non-stationary reinforcement learning takes into account the dynamics and uncertainty of network attack behaviors, so it can attract and respond to these attacks more effectively.

[0077] Table 5 The number and distribution of attack requests collected by different honeypots

[0078] RUQL Honeypot QL Honeypot Random Honeypot Exploitation 118 19 0 Permission bypass 10 10 0 Brute force 227 130 13 Malicious Scan 276 139 21 Access attempt 369 163 54

[0079] Figure 1 It is a flow chart of the method of the present invention. There are three main components that run separately, but share data with each other during the learning process. The response database mainly stores the response data obtained about the industrial Internet of Things devices. The data receiving module is used to capture the attack request and simplify the request state using the LDA algorithm. On the other hand, the request can be used for later analysis of the attack trajectory, guessing the attacker's intention, and evaluating the analysis algorithm model. The adversarial interaction module uses the RUQL algorithm to train the model based on the given response feedback of the attacker. After several rounds of learning iterations, the honeypot of the present invention can optimize the model to reply to the attacker. From the beginning, the honeypot system of the present invention behaves like a low-interaction honeypot because the system starts with zero knowledge of the industrial Internet of Things devices and their behaviors. According to experimental evaluation, the industrial Internet of Things honeypot based on adversarial reinforcement learning can continuously learn and adapt to changes in network attacks when dealing with attackers with complex and changeable behaviors, and dynamically adjust the behavior and strategy of the honeypot according to the observed attack behavior to more effectively attract and respond to attacks.

[0080] Figure 2This is a schematic diagram of the LDA analysis method based on the multimodal request data of the Industrial Internet of Things. The LDA model can be used to perform cluster analysis on the device request data. This is an innovative attempt, because LDA is usually used for the analysis of text data, and applying it to the Industrial Internet of Things device request data can dig out the potential topics and patterns behind the device requests. Specifically, LDA first uses predefined delimiters (such as commas, spaces, etc.) to split each document (request) into a series of words (usually features or tags). Next, the words of all documents (requests) are combined to form a global corpus, which is also called a dictionary and contains the vocabulary in all documents. Then the frequency of occurrence (statistical distribution) of each word in the corpus is calculated. Select a number of topics n (this is usually a hyperparameter that needs to be determined through experiments or domain knowledge). Use the LDA algorithm to assign words to n topics. Finally, after the LDA model is trained, new requests (documents) can be input into the model and see which topic they are most related to. The LDA implementation is mainly based on two core libraries: the jieba word segmentation library and the LDA model library in Gensim. The specific steps are as follows. First, before topic modeling, the text needs to be preprocessed, including word segmentation, removal of stop words and punctuation, etc. Word segmentation can be performed using tools such as jieba, and stop word removal can be performed using the nltk library. After word segmentation and dictionary construction, a document-word frequency matrix is ​​generated through the Gensim library. This matrix counts the frequency of occurrence of each word in each document based on the words in the dictionary. Next, the LdaModel class in the Gensim library is used to run the LDA model for further topic analysis. The number of topics, dictionary, and document-word frequency matrix need to be specified as input parameters. The model will automatically learn the distribution of topics and words. Once the model training is complete, the show_topics method can be used to extract topics. Each topic is represented by a set of high-weight words.

[0081] Figure 3It is the construction of the MDP (Markov Decision Process) model in a real adversarial scenario instance provided by an embodiment of the present invention. Since the entire graph is too large, the embodiment of the present invention only emphasizes the specific vulnerabilities and attack behaviors that are exploited. The rectangular box in the figure represents the state, the elliptical box represents the action, and the triangular box represents the action that can be taken in a specific state. In our context, each state is a unique request abstraction, and each operation of the request is a unique response candidate to be replied. The actions that can be selected under the root path ' / ' are responseA, responseB, and responseC. ResponseA is an invalid response, which may cause the attacker to end the session prematurely. ResponseB and responseC represent different types of valid responses, respectively. It can be seen from the figure that when the RUQL honeypot replies to responseA, the session ends directly; when the RUQL honeypot replies to responseB, the attacker can send ' / image / lgbg.jpg' or ' / favicon.ico' for pre-attack checks. The ' / image / lgbg.jpg' node represents a certain image information of the device. By analyzing

[0082] / lgbg.jpg to identify this as a Hikvision camera. The node with ' / favicon.ico' indicates that the attacker is trying to access the favicon file, which represents the manufacturer's small icon; when the RUQL honeypot replies with responseC, after completing the pre-attack check, it can guide the attacker to the vulnerable transition node ' / Security / users', and then enter the specific vulnerability node 'CVE-2021-36260'. Figure 3 It can be seen that the RUQL-based agent in the adversarial environment not only regards response C as a valid response, but also regards response B as a valid response, and response B can effectively avoid the pre-attack check. Thus, it guides the attacker to access the vulnerable transition node ' / Security / users'.

[0083] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. The step numbers in the specification and claims are only for the convenience of clearly describing the technical solution of the present invention, and the sequence of the numbers is not limited. Obviously, for professionals in this field, after understanding the content and principle of the present invention, it is possible to make various modifications and changes in form and details without departing from the principle and structure of the present invention, but these modifications and changes based on the idea of ​​the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A method for deploying industrial Internet of Things honeypots based on adversarial reinforcement learning, characterized in that: include: (1) Collect IIoT device data, including IIoT device IP, IIoT device request data, and IIoT device response data; (2) Based on the IIoT device request data collected in step (1), the LDA analysis method based on IIoT multimodal request data is used to perform data clustering and classify the request data into different categories; (3) Based on the classified request data in step (2), a Markov decision model is used to construct an interaction model between the honeypot and the attacker, where the honeypot is the intelligent agent, the network attacker attacking the honeypot is the environment, the state is the classified network attacker's request, the action is the response data replied by the honeypot, and the request to induce attack and extend the session is rewarded; (4) Based on the Markov decision model constructed in step (3), the adversarial reinforcement learning algorithm RUQL is used to train the model. The honeypot selects the optimal response to respond to attackers with complex and changeable behaviors.

2. According to claim 1, the industrial Internet of Things honeypot deployment method based on adversarial reinforcement learning is characterized in that: The collecting of industrial Internet of Things device data specifically includes: collecting real industrial Internet of Things device request data and IP; traversing the IP to open the TCP socket, actively sending TCP requests to the industrial Internet of Things devices, observing and collecting their response data and storing them in the response database.

3. The industrial Internet of Things honeypot deployment method based on adversarial reinforcement learning according to claim 1 is characterized in that: In step (1), the method of collecting request data is: deploying a low-interaction honeypot instance to monitor the bait port to collect request data, or using the penetration tool Burpsuit to collect request data.

4. The industrial Internet of Things honeypot deployment method based on adversarial reinforcement learning according to claim 1 is characterized in that: Step (2) is specifically as follows: each IIoT device request is regarded as a document, and the predefined request data category is regarded as the topic of the document; each document is divided into a series of words by a predefined separator; the words of all documents are combined together to form a global corpus; Calculate the frequency of occurrence of each word in the corpus; take the number of predefined request data categories as the number of topics, and after training the LDA model, you can get the input request data categories.

5. The industrial Internet of Things honeypot deployment method based on adversarial reinforcement learning according to claim 1 is characterized in that: The categories of the requested data include: attack data targeting industrial Internet of Things devices, attack data containing Linux Bash instructions, attack data of malware upload type, other attack data, and termination status data.

6. The industrial Internet of Things honeypot deployment method based on adversarial reinforcement learning according to claim 1 is characterized in that: Step (2) also includes a data preprocessing step, specifically: removing invalid or erroneous data records and standardizing the request data so that the mean of each feature is 0 and the standard deviation is 1.

7. The industrial Internet of Things honeypot deployment method based on adversarial reinforcement learning according to claim 1 is characterized in that: In step (3), the goal of state definition is to capture the key features of the interaction between the attacker and the honeypot while keeping the state space relatively compact; the states of the interaction between the attacker and the honeypot are divided into five state sets, namely S = {s1, s2, s3, s4, s5}; s1 is the attack data against industrial IoT devices, including vulnerability exploitation, permission bypass, brute force cracking, malicious scanning, and access attempts; s2 is the attack data containing Linux Bash instructions; s3 is the attack data of malware upload type, and this type of attack is identified by interpreting the file suffix; s4 is other attack data, which refers to requests that cannot be processed, including TLS communications composed of hexadecimal data; s5 is the termination state data, and when the agent does not receive the next state or times out, it will automatically enter the termination state.

8. The industrial Internet of Things honeypot deployment method based on adversarial reinforcement learning according to claim 1 is characterized in that: In step (4), the RUQL algorithm considers the difference between the actual return and the predicted Q value and adjusts the update according to the difference between the strategy and the behavior. The Q value update formula is: Among them, the expression Measures the difference between the current Q-value prediction and the actual Q-value, r is the reward obtained after performing action a in state s; γ is the discount factor; is in the next state s ′ The maximum Q value of all possible actions in state s; α is the learning rate; π(s,a) is the probability that the strategy chooses action a in state s, is a weight used to adjust the update to reflect the difference between the policy and the behavior; β is a tuning parameter used to prevent the policy from approaching zero.

9. The industrial Internet of Things honeypot deployment method based on adversarial reinforcement learning according to claim 8 is characterized in that: In step (4), the specific operation steps of the RUQL algorithm are: a. Observe the current state s and arbitrarily initialize the value function Q; b. For each time step t, the agent selects action a according to the current value function Q and exploration strategy, outputs action a and observes a result reward r and the next state s of the environment ′ ; Update the value function Q using the Q value update formula in RUQL; repeat step b until the value function Q converges.

10. An industrial Internet of Things honeypot deployment device implemented by the method according to any one of claims 1 to 9, characterized in that: It consists of three components, which run independently and share data with each other during the learning process; specifically: A response database for storing the obtained response data about the industrial Internet of Things devices; The data receiving module is used to capture attack requests and simplify the requests using the LDA algorithm; The adversarial interaction module uses the RUQL algorithm to train the model based on the given response feedback of the attacker. When dealing with attackers with complex and changeable behaviors, the honeypot continuously learns and adapts to the changes in network attacks, and dynamically adjusts the behavior and strategy of the honeypot based on the observed attack behavior to more effectively attract and respond to attacks.

Images