Reinforcement learning honeypot construction method and device based on BERT model

Through the reinforcement learning method based on the BERT model, the problem of building heterogeneity and high-interaction honeypots in IoT devices is solved, and the high-interaction honeypot with automatic learning and adaptability is realized, which improves the collection effect of IoT network attack data.

CN117834228BActive Publication Date: 2024-11-26ZHEJIANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311793754.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-25
Publication Date
2024-11-26
Estimated Expiration
2043-12-25

AI Technical Summary

Technical Problem

The heterogeneity of IoT devices and the existing high-interaction honeypot construction methods are difficult to comprehensively collect IoT network attack data, which limits the deception and data collection effects of honeypots.

Method used

Using reinforcement learning method based on BERT model, we can collect HTTP network interaction data by deploying Internet of Things honeypots, use the ATT&CK framework to identify attack data, build attack chains, extract features, update BERT models, train attack category classification models, and use Markov to The decision model and Q-learning algorithm are trained online, selecting the optimal response strategy to trick cyber attackers.

Benefits of technology

It realizes automatic learning of IoT device behavior knowledge, adaptively constructs heterogeneous high-interactive honeypots, improves the honeypot's deception ability, and can effectively collect detailed data on all behaviors of IoT network attackers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117834228B_ABST
    Figure CN117834228B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for constructing a reinforcement learning honeypot based on a BERT model. The method comprises: deploying an Internet of Things honeypot to collect HTTP network interaction data, and associating network attack data with ATT&CK attack strategies to form an attack chain; constructing more similar attack categories based on the main text features of the network attack data to fine-tune the BERT model; finally, using a trained attack category classification model to perform category recognition of network attack data, and using the result of category recognition as a state variable in a Markov decision process to construct a reinforcement learning honeypot. The method of the present invention can automatically learn the behavioral knowledge of Internet of Things devices, adaptively construct a heterogeneous high-interaction honeypot for the Internet of Things, and effectively improve the honeypot's deception ability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of network security and relates to a method and device for constructing a reinforcement learning honeypot based on a BERT model. Background Art

[0002] In recent years, IoT technology has been continuously accumulated and upgraded, and more and more physical devices, sensors and other devices that can collect data and communicate are connected to the Internet, and the number of IoT devices in the world has grown rapidly. However, the common logical defects and security vulnerabilities in the firmware of IoT devices have not been effectively resolved, making many IoT devices potential targets for attackers.

[0003] Honeypot is an active defense technology for security threats. It delays attacks and protects real devices by simulating one or more vulnerable services or devices. At the same time, honeypot can actively attract attackers and capture network attack data in order to discover network threats and extract threat features. This intelligence helps the defender understand the attacker's intentions, organization, and the threat actions behind them, so as to take corresponding countermeasures.

[0004] Honeypots can be classified according to the level of interaction they have with network attackers: low-interaction honeypots, medium-interaction honeypots, and high-interaction honeypots. Among them, low-interaction honeypots only simulate one or more simple services to record some information about the activities of network attackers. Low-interaction honeypots have the advantages of easy deployment, low risk, and low cost. However, low-interaction honeypots have a low degree of simulation, and network attackers can easily identify their existence. This feature limits the effectiveness of low-interaction honeypots in attracting and capturing network attackers, and can only collect limited network attack data. Medium-interaction honeypots will provide limited responses to network attackers based on service simulation, which can deceive attackers to a certain extent and lure them to send more network attack data. High-interaction honeypots simulate the entire system and service, which can effectively lure network attackers and collect detailed data on all behaviors of network attackers.

[0005] The collection of network attack information by high-interaction honeypots can help security practitioners better understand the IoT attack ecosystem and formulate corresponding defense measures. However, there are currently the following problems in the construction of IoT high-interaction honeypots: On the one hand, due to the heterogeneity of IoT devices, it is difficult to simulate different types of IoT devices from different vendors through software. On the other hand, the construction method of IoT high-interaction honeypots based on real devices is limited to specific devices, making it difficult to comprehensively collect IoT network attack data. Summary of the invention

[0006] The purpose of the present invention is to provide a method and device for constructing a reinforcement learning honeypot based on the BERT model in view of the deficiencies and shortcomings in the current construction of heterogeneous high-interaction honeypots for the Internet of Things.

[0007] The objective of the present invention is achieved through the following technical solutions:

[0008] According to a first aspect of this specification, a method for constructing a reinforcement learning honeypot based on a BERT model is provided, the method comprising the following steps:

[0009] (1) deploying an IoT honeypot to collect HTTP-based network interaction data, wherein the interaction data includes response data and attack data; the attack data is data sent to the IoT honeypot by a network attacker, and the response data is obtained by the IoT honeypot sending a security request to the IoT device;

[0010] (2) Based on the attack data sent by the network attacker in step (1), the attack data from the same network attacker within the session time is categorized using the attack strategy in the ATT&CK framework, and the attack chain is constructed using the categorized results;

[0011] (3) Based on the attack data identified in step (2), feature screening and extraction are performed on the attack data of different categories, main features of the attack data of different categories are extracted, and the main features are used to construct attack data of different categories;

[0012] (4) Based on the different categories of attack data constructed in step (3), segment and filter different words in the attack data to supplement the vocab.txt file in the BERT model;

[0013] (5) Based on the vocab.txt file updated in step (4) and the different categories of attack data constructed in step (3), the pre-trained BERT model is used to fine-tune the downstream tasks and train the attack category classification model; the attack category classification model is used to perform real-time classification and prediction of the attack data in the network collected by the IoT reinforcement learning honeypot;

[0014] (6) Based on the attack data classification prediction result obtained in step (5), the result is used as a state variable to model the interaction between the reinforcement learning honeypot and the network attacker as a Markov decision model, and the Q-learning algorithm is used to perform online training of the Markov decision model;

[0015] (7) Based on the Markov decision model trained in step (6), the reinforcement learning honeypot selects the optimal response to reply to different IoT attack data.

[0016] Furthermore, in step (1), an IoT honeypot is deployed on a cloud server to collect network interaction data of IoT devices based on the HTTP protocol; the security request sent by the IoT honeypot to the IoT device comes from the security data after the attack data is filtered. During the filtering process, the shell instructions in the attack data are detected by regularized matching keywords. Once the shell instructions are successfully detected, the HTTP attack data will not be sent to the IoT device.

[0017] Furthermore, in step (2), based on the attack strategies in the ATT&CK framework, different attack data are marked with attack category labels, and different categories correspond to different stages of network security incidents; based on the attack data sent to the IoT honeypot by the same network attacker, an attack chain is constructed, and different stages on the attack chain correspond to different attack strategies of the ATT&CK framework.

[0018] Furthermore, in step (3), firstly, the Accept, Accept-Encoding, Accept-Language, Host, Connection fields and their field values ​​in the request header of the attack data are deleted, and the remaining attack data is retained; then, the main features of the attack data under different categories are extracted, and text construction based on the main features is performed, and the construction method includes a random combination of basic features, including the request method, request path, protocol version and User-Agent in the request header, and adding specific category features related to the attack category in the request body; such a combination constitutes attack data of different categories.

[0019] Furthermore, in step (4), in order to supplement the vocab.txt file of the BERT model, the attack data is first symbolically segmented, and high-frequency words are counted to form an HTTP attack data vocabulary; then, the vocabulary is compared with the vocab.txt file to filter out the missing HTTP attack data words, which are used to update the vocab.txt file to optimize the embedding representation of the BERT model.

[0020] Furthermore, in step (5), a pre-trained English BERT model is used to train an attack category classification model, wherein the attack category classification model includes an input layer, an embedding layer, a multi-layer encoder, and an output layer; the input layer is used to receive attack data, and the embedding layer is used to convert words in the attack data into corresponding vector representations; each layer of the multi-layer encoder includes a self-attention mechanism and a feedforward neural network, which are used to capture the contextual relationship between words, and its input is a vector representation of the words, and its output is a vector containing the contextual relationship of the words; the output layer is a fully connected layer, which is used to output the probabilities of different attack categories.

[0021] Furthermore, in step (6), the interaction process between the network attacker and the reinforcement learning honeypot is modeled as a Markov decision model, and the Q-learning algorithm is used to solve the optimal strategy; the reinforcement learning honeypot is the intelligent agent in the Markov decision process, the external network environment is the environment, and the attack data sent by the network attacker is the state; the response of the reinforcement learning honeypot to different attack data is the action.

[0022] Furthermore, in step (6), the state transfer matrix is ​​obtained by the transfer of each attack stage on all attack chains in step (2) and the corresponding response data statistics. Each element in the matrix represents the probability that the attack data is transferred to the next stage of the attack chain after the reinforcement learning honeypot responds to the attack data of the current stage of the attack chain. The probability is calculated by dividing the number of times the attack data of the current stage is transferred to the attack data of the next specific stage by the total number of times the attack data of the current stage is transferred to all other next stages.

[0023] Furthermore, in step (7), the reinforcement learning honeypot selects the optimal response strategy according to the training results of the Markov decision model. The strategy optimally selects the response reply for the heterogeneous HTTP attack data from the IoT devices to better lure the network attackers and achieve high interactive interaction with the network attackers.

[0024] According to the second aspect of this specification, a device for constructing a reinforcement learning honeypot based on a BERT model is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the method for constructing a reinforcement learning honeypot based on the BERT model as described in the first aspect.

[0025] According to a second aspect of this specification, a computer-readable storage medium is provided, on which a program is stored. When the program is executed by a processor, it is used to implement the reinforcement learning honeypot construction method based on the BERT model as described in the first aspect.

[0026] The beneficial effects of the present invention are as follows: the present invention first deploys an IoT honeypot to collect HTTP network interaction data, and associates the network attack data with the ATT&CK attack strategy to form an attack chain; then constructs more similar attack categories based on the main text features of the network attack data to fine-tune the BERT model; finally, uses the trained attack category classification model to perform category recognition of the network attack data, and uses the result of category recognition as a state variable in the Markov decision process to construct a reinforcement learning honeypot. The method of the present invention can automatically learn the behavioral knowledge of IoT devices, adaptively construct heterogeneous high-interaction honeypots for the IoT, and effectively improve the honeypot's deception ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a method for constructing a reinforcement learning honeypot based on a BERT model provided by an embodiment of the present invention;

[0028] Figure 2 It is a schematic diagram of the interaction process between the reinforcement learning honeypot and the network attacker provided by an embodiment of the present invention;

[0029] Figure 3 It is a structural diagram of a reinforcement learning honeypot construction device based on a BERT model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] This embodiment provides a method for constructing a reinforcement learning honeypot based on a BERT model, comprising the following steps:

[0032] (1) An IoT honeypot is deployed on a cloud server to collect network interaction data of IoT devices based on the HTTP protocol, including HTTP response data and HTTP attack data. HTTP attack data is data sent to the IoT honeypot by a network attacker. HTTP response data is obtained by the IoT honeypot sending a security request to the IoT device, and the security request comes from the security data after the HTTP attack data is filtered. During the filtering process, shell instructions in the HTTP attack data are detected by regularized matching keywords. Once the shell instruction is successfully detected, the HTTP attack data will not be sent to the IoT device to collect HTTP response data. The HTTP response data successfully collected to the IoT honeypot will be used as the response resource of the reinforcement learning honeypot. When the reinforcement learning honeypot interacts with the network attacker online, it is used to respond to the HTTP attack data from the network attacker.

[0033] (2) Based on the HTTP attack data collected in step (1), first determine whether it comes from the same IP address to distinguish different network attackers. For network attackers from the same IP address, they are regarded as the same network attacker. Then set a session time threshold for the HTTP attack data belonging to the same network attacker. The threshold represents the time interval between two adjacent attacks by the same network attacker. Finally, the HTTP attack data from the same network attacker within the session time is regarded as an attack chain, and the HTTP attack data on the attack chain is labeled with attack category labels based on the attack strategy in the ATT&CK framework. There are 12 categories of category labels, corresponding to the 12 stages of network security incidents. A typical attack chain example is as follows: initial access → credential access → defense bypass → data collection → command and control. For the same network attacker who has sent HTTP attack data, if new HTTP attack data is sent again after the session time, the two attacks are no longer regarded as different stages in the same attack chain.

[0034] (3) Based on the different categories of HTTP attack data marked in step (2), first perform feature screening on them, specifically: delete some features of HTTP attack data under all categories, mainly the Accept, Accept-Encoding, Accept-Language, Host, Connection fields and their field values ​​in the request header. Then extract the main features of HTTP attack data under different categories, and construct HTTP attack data under different attack categories based on the main features. The main features include the following basic features: request method, request path, HTTP protocol version, User-Agent in the request header, and category features related to the attack category in the request body. Category features are divided into two categories, one is the shell instructions that match the attack category, and the other is the keywords that match the attack category. When constructing HTTP attack data of different categories, the random combination of basic features, plus the category features of specific categories, together form a complete, specific category of HTTP attack data. These constructed HTTP attack data of different categories will be used for the downstream task fine-tuning of the BERT model.

[0035] (4) Based on the different types of HTTP attack data constructed in step (3), the vocab.txt file of the BERT model is supplemented. First, the HTTP attack data is segmented according to the symbols in it, and the high-frequency words after segmentation are counted to form an HTTP attack data vocabulary. Then the HTTP attack data vocabulary is compared with the vocab.txt file to filter out the missing HTTP attack data words in the vocab.txt file. These missing words will be used to update the vocab.txt file to better obtain the embedding representation of the BERT model.

[0036] (5) Based on the vocab.txt file updated in step (4) and the different categories of HTTP attack data constructed in step (3), the pre-trained BERT model is used to fine-tune the downstream task, namely the attack category classification task, to establish an attack category classification model. The attack category classification model includes an input layer, an embedding layer, a multi-layer encoder, and an output layer. The input layer is designed to receive HTTP attack data. The task of the embedding layer is to convert the vocabulary in the HTTP attack data into corresponding vector representations. The multi-layer encoder consists of 12 layers, each of which contains a self-attention mechanism and a feedforward neural network. The multi-layer encoder is designed to capture the contextual relationship between words, and its input is the vector representation of the vocabulary. Finally, the last layer of the multi-layer encoder outputs a vector containing the contextual relationship of the vocabulary. The output layer is a fully connected layer that receives the output vector from the multi-layer encoder. The output layer is responsible for classifying the output vector and outputting the probabilities of different attack categories. Through the continuous training of the attack category classification model, the real-time HTTP attack data collected by the reinforcement learning honeypot can be classified and predicted.

[0037] (6) Based on the HTTP attack data classification prediction results obtained in step (5), when the reinforcement learning honeypot is running online, the reinforcement learning honeypot is modeled using a Markov decision model, and the interaction process between the reinforcement learning honeypot and the network attacker is modeled as a Markov decision process. In the present invention, the reinforcement learning honeypot is an agent in the Markov decision process; the external network environment is the environment; the HTTP attack data sent by the network attacker is the state; the response of the reinforcement learning honeypot to different HTTP attack data is the action; the state transfer matrix is ​​obtained by the transfer of each attack stage on all attack chains in step (2) and the corresponding response data statistics. Each element in the matrix represents the probability that the HTTP attack data is transferred to the next stage of the attack chain after the reinforcement learning honeypot responds to the HTTP attack data of the current stage of the attack chain. Specifically, the probability is calculated by dividing the number of times the HTTP attack data of the current stage is transferred to the HTTP attack data of the next specific stage by the total number of times the HTTP attack data of the current stage is transferred to all other next stages. The reward function can be divided into three different cases:

[0038]

[0039] In the above formula, s t Represents the HTTP attack data from the network attacker at the current stage of the attack chain, s t+1 Represents the HTTP attack data from the same network attacker in the next stage of the attack chain, a tRepresents the reinforcement learning honeypot receiving the HTTP attack data s at the current stage of the attack chain t The response data to the network attacker, allow means that the response data is allowed to reply, r a (s t ,a t ) represents the reinforcement learning honeypot in HTTP attack data s t Reply to the attacker with data a t In the online interaction between the reinforcement learning honeypot and the network attacker, when the reinforcement learning honeypot is in a state s t Take action a t , and upon receiving the next state s t+1 , it means that the HTTP attack data category has been transferred from the current stage to the next stage of the attack chain, and the reward value is 2. r Take action a t , and the received HTTP attack data is still s t , that is, if the HTTP attack data from two adjacent stages from the same network attacker belong to the same category, then the reward value is 1. In other cases, the reward value is 0.

[0040] (7) Based on the Markov decision model constructed in step (6), the Q-learning algorithm is used to perform online training on the reinforcement learning honeypot. When the network attacker sends HTTP attack data, the reinforcement learning honeypot can select multiple response strategies. After a specific response is successfully returned to the network attacker and new HTTP attack data is received, the reinforcement learning honeypot calculates the reward of the interaction process based on the Bellman equation and updates the Q value. Through the continuous iteration of the Q value of the reinforcement learning honeypot, a heterogeneous high-interaction honeypot can be trained, which can automatically select the optimal response to reply to HTTP attack data.

[0041] Figure 1: is the flow chart of the present invention. The process is mainly divided into two stages. First, in the first stage, the HTTP attack data collected by the IoT honeypot is preprocessed. This preprocessing includes category labeling, feature screening and data construction of the original HTTP attack data. The category labeling is based on the knowledge of the ATT&CK attack framework, and the original HTTP attack data is labeled with attack category classification. Then, by analyzing the characteristics of the data, the same category attack data based on the main features is constructed. During the training process of the attack category classification model, the BERT model is used to train downstream tasks, and the trained model is used to perform real-time category prediction of HTTP attack data from network attackers. In the second stage, the output of the attack category classification model is used as the state, combined with the response data collected by the IoT honeypot in the first stage and the extracted attack chain, a reinforcement learning honeypot is constructed. The honeypot automatically learns the behavioral knowledge of IoT devices to meet the needs of heterogeneous high interaction. With the continuous learning iteration of the reinforcement learning honeypot, it can return the optimal response value for different HTTP attack data, thereby effectively deceiving network attackers and collecting detailed data on all behaviors of network attackers.

[0042] Figure 2 is a specific Markov decision process structure. In the continuous interaction between the reinforcement learning honeypot and the network attacker, when the reinforcement learning honeypot obtains HTTP attack data s from the network attacker t , the reinforcement learning honeypot will return a response a based on the data t The attacker will then send new HTTP attack data s according to the response returned by the reinforcement learning honeypot. t+1 To the reinforcement learning honeypot. At this point, the reinforcement learning honeypot will receive a reward value R t+1 The goal of the reinforcement learning honeypot is to obtain the maximum cumulative reward during the interaction with the network attacker in order to train the optimal response return strategy.

[0043] Corresponding to the aforementioned embodiment of the method for constructing a reinforcement learning honeypot based on the BERT model, the present invention also provides an embodiment of a device for constructing a reinforcement learning honeypot based on the BERT model.

[0044] See also Figure 3 , this embodiment provides a reinforcement learning honeypot construction device based on the BERT model, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the reinforcement learning honeypot construction method based on the BERT model in the above embodiment.

[0045] The embodiment of the reinforcement learning honeypot construction device based on the BERT model of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 3 As shown, it is a hardware structure diagram of any device with data processing capability where the reinforcement learning honeypot construction device based on the BERT model of the present invention is located, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0046] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0047] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0048] An embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the method for constructing a reinforcement learning honeypot based on the BERT model in the above embodiment is implemented.

[0049] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0050] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0051] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a reinforcement learning honeypot based on the BERT model, characterized in that: The steps include: (1) deploying an IoT honeypot to collect HTTP-based network interaction data, wherein the interaction data includes response data and attack data; the attack data is data sent to the IoT honeypot by a network attacker, and the response data is obtained by the IoT honeypot sending a security request to the IoT device; (2) Based on the attack data sent by the network attacker in step (1), the attack data from the same network attacker within the session time is categorized using the attack strategy in the ATT&CK framework, and the attack chain is constructed using the categorized results; (3) Based on the attack data identified in step (2), feature screening and extraction are performed on the attack data of different categories, and the main features of the attack data of different categories are extracted, and the main features are used to construct the attack data of different categories; specifically: First, the Accept, Accept-Encoding, Accept-Language, Host, Connection fields and their field values ​​in the request header of the attack data are deleted, and the remaining attack data is retained; then the main features of the attack data under different categories are extracted, and text construction based on the main features is performed. The construction method includes a random combination of basic features, including the request method, request path, protocol version, and User-Agent in the request header, and the specific category features related to the attack category in the request body; such a combination constitutes attack data of different categories; (4) Based on the different categories of attack data constructed in step (3), segment and filter different words in the attack data to supplement the vocab.txt file in the BERT model; (5) Based on the vocab.txt file updated in step (4) and the different categories of attack data constructed in step (3), the pre-trained BERT model is used to fine-tune the downstream tasks and train the attack category classification model; the attack category classification model is used to perform real-time classification and prediction of the attack data in the network collected by the IoT reinforcement learning honeypot; (6) Based on the attack data classification prediction result obtained in step (5), the result is used as a state variable to model the interaction between the reinforcement learning honeypot and the network attacker as a Markov decision model, and the Q-learning algorithm is used to perform online training of the Markov decision model; (7) Based on the Markov decision model trained in step (6), the reinforcement learning honeypot selects the optimal response to reply to different IoT attack data.

2. According to a method for constructing a reinforcement learning honeypot based on a BERT model according to claim 1, it is characterized in that: In step (1), an IoT honeypot is deployed on a cloud server to collect network interaction data of IoT devices based on the HTTP protocol; the security request sent by the IoT honeypot to the IoT device comes from the security data after the attack data is filtered. During the filtering process, the shell instructions in the attack data are detected by regularized matching keywords. Once the shell instructions are successfully detected, the attack data will not be sent to the IoT device.

3. According to a method for constructing a reinforcement learning honeypot based on a BERT model according to claim 1, it is characterized in that: In step (2), based on the attack strategies in the ATT&CK framework, different attack data are marked with attack category labels, and different categories correspond to different stages of network security incidents; based on the attack data sent to the IoT honeypot by the same network attacker, an attack chain is constructed, and different stages on the attack chain correspond to different attack strategies of the ATT&CK framework.

4. The method for constructing a reinforcement learning honeypot based on a BERT model according to claim 1, characterized in that: In step (4), in order to supplement the vocab.txt file of the BERT model, the attack data is first symbolically segmented and high-frequency words are counted to form an HTTP attack data vocabulary. Subsequently, the vocabulary is compared with the vocab.txt file to filter out the missing HTTP attack data words and use them to update the vocab.txt file to optimize the embedding representation of the BERT model.

5. The method for constructing a reinforcement learning honeypot based on a BERT model according to claim 1, characterized in that: In step (5), a pre-trained English BERT model is used to train an attack category classification model, wherein the attack category classification model includes an input layer, an embedding layer, a multi-layer encoder, and an output layer; the input layer is used to receive attack data, and the embedding layer is used to convert words in the attack data into corresponding vector representations; each layer of the multi-layer encoder includes a self-attention mechanism and a feedforward neural network, which are used to capture the contextual relationship between words, and its input is a vector representation of the words, and its output is a vector containing the contextual relationship of the words; the output layer is a fully connected layer, which is used to output the probabilities of different attack categories.

6. The method for constructing a reinforcement learning honeypot based on a BERT model according to claim 1, characterized in that: In step (6), the interaction process between the network attacker and the reinforcement learning honeypot is modeled as a Markov decision model, and the Q-learning algorithm is used to solve the optimal strategy; The reinforcement learning honeypot is an intelligent agent in the Markov decision process, the external network environment is the environment, and the attack data sent by the network attacker is the state; The response of the reinforcement learning honeypot to different attack data is action.

7. The method for constructing a reinforcement learning honeypot based on a BERT model according to claim 6, characterized in that: In step (6), the state transfer matrix is ​​obtained by the transfer of each attack stage on all attack chains in step (2) and the corresponding response data statistics. Each element in the state transfer matrix represents the probability that the attack data is transferred to the next stage of the attack chain after the reinforcement learning honeypot responds to the attack data of the current stage of the attack chain. The probability is calculated by dividing the number of times the attack data of the current stage is transferred to the attack data of the next specific stage by the total number of times the attack data of the current stage is transferred to all other next stages.

8. The method for constructing a reinforcement learning honeypot based on a BERT model according to claim 1, characterized in that: In step (7), the reinforcement learning honeypot selects the optimal response strategy based on the training results of the Markov decision model. The strategy optimally selects the response reply for the heterogeneous HTTP attack data from IoT devices to better lure network attackers and achieve high interactive interaction with network attackers.

9. A device for constructing a reinforcement learning honeypot based on a BERT model, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it is used to implement the reinforcement learning honeypot construction method based on the BERT model as described in any one of claims 1 to 8.