Malicious traffic detection anti-attack method and device

By combining traffic semantic model and deep reinforcement learning technology, the generation of anti-malware traffic that can escape detection can solve the robustness of malicious traffic detection systems in the existing technology when facing zero-day attacks and black box anti-malware attacks, and improves the Network security protection level.

CN119071023AActive Publication Date: 2024-12-03TSINGHUA UNIVERSITY
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202411058579.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-12-03
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

The existing malicious traffic detection system is limited in its effectiveness in the face of zero-day attacks and black-box adversarial attacks, and lacks effective robustness evaluation methods.

Method used

A black box adversarial attack method based on traffic semantic model and deep reinforcement learning is proposed. By extracting feature sequences, training traffic semantic model, generating adversarial feature sequences and combining punishment mechanisms, it generates adversarial malicious intentions that can escape detection and maintain the original traffic properties. flow.

Benefits of technology

Effectively generate practical and anti-malicious traffic, improves the robustness of the malicious traffic detection system, and helps identify potential weak links, improving the level of network security protection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071023A_ABST
    Figure CN119071023A_ABST
Patent Text Reader

Abstract

The invention discloses a malicious traffic detection anti-attack method and device. The method comprises the following steps: taking a first feature sequence as an input sequence of an initialized traffic semantic model; randomly shielding a partial sequence in the input sequence, and predicting an original value of the shielded partial sequence to train a traffic semantic model; processing the extracted second feature sequence by using a deep reinforcement learning model to obtain action data, and processing the second feature sequence based on the action data and a trained traffic semantic model to obtain a confrontation feature sequence; and converting the adversarial feature sequence into adversarial malicious traffic, sending the adversarial malicious traffic from the sending end to the receiving end, obtaining feedback information according to the number of data packets successfully received by the receiving end, combining the feedback information with a preset punishment mechanism, and optimizing the network of the deep reinforcement learning model to obtain a trained deep reinforcement learning model. According to the method, the anti-malicious traffic which not only can be detected in an escape manner, but also can keep the original traffic property can be generated under an unknown condition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of cyberspace security and artificial intelligence security technology, and in particular to a method and device for detecting malicious traffic and countering attacks. Background Art

[0002] Malicious traffic detection technology is a key research direction in the field of network security, which is mainly divided into signature-based malicious traffic detection systems and machine learning-based malicious traffic detection systems. Currently, signature-based systems rely on predefined malicious feature libraries for traffic analysis. Although they perform well in detecting known threats, this method is limited in effectiveness when facing new traffic attack modes such as zero-day attacks. In contrast, machine learning-based detection systems analyze traffic behavior patterns and use machine learning technology to automatically identify unknown or mutated malicious traffic, showing higher flexibility and adaptability.

[0003] In addition, adversarial attack is an attack technique that confuses the model output by interacting with the machine learning model and applying small perturbations to the input data. At present, adversarial attacks can be divided into two types: white-box attacks and black-box attacks. White-box attacks assume that the attacker fully understands the internal structure and parameters of the target model, while black-box attacks are performed when the attacker only knows the input and output of the model. It can be seen that black-box adversarial attacks are more practical and important in network security research because they are closer to actual attack scenarios.

[0004] At the same time, the generative model is a type of machine learning model that automatically generates data by learning a large amount of data distribution. In the field of natural language processing, BERT (Bidirectional Encoder Representations from Transformers) is an advanced generative model that captures the deep semantic relationship of sequence data through a specific multi-layer bidirectional encoder structure.

[0005] Furthermore, deep reinforcement learning has been applied to many fields such as cyberspace security, next-generation Internet, robotics, autonomous driving, and medical care. In general, reinforcement learning methods aim to solve the problem of how intelligent agents interact with the environment to maximize cumulative rewards. These problems can usually be modeled as a Markov decision process MDP = (S, A, P, R, T). Among them, S represents the state space of the agent, that is, the set of possible states of the agent in the environment; A represents the action space of the agent, that is, the set of optional actions of the agent; P represents the state transition function, which is used to describe the probability distribution of the next state that the agent may reach after performing an action; R represents the reward function, that is, the benefit obtained by the agent from the environment after taking an action; T represents the time range of the decision-making process, which is used to describe the termination condition of the decision. Deep reinforcement learning combines deep learning technology with reinforcement learning, and the agent uses neural networks to make decisions.

[0006] Currently, there is an urgent need for a method to implement adversarial attacks against malicious traffic detection systems in black-box scenarios in order to evaluate the robustness of malicious traffic detection systems. Summary of the invention

[0007] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0008] To this end, the present invention proposes a malicious traffic detection counterattack method, which aims to generate counter-malicious traffic that can escape detection and maintain its original traffic properties without knowing the specific parameters and implementation details of the target detection system.

[0009] Another object of the present invention is to provide a malicious traffic detection and counterattack device.

[0010] To achieve the above object, an embodiment of the present invention provides a method for detecting malicious traffic and resisting attacks, including:

[0011] Extracting a first feature sequence, and using the first feature sequence as an input sequence of an initialized traffic semantic model;

[0012] Randomly masking part of the sequence in the input sequence, and using the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model;

[0013] Using a deep reinforcement learning model to process the extracted second feature sequence to obtain action data, and based on the action data and the trained traffic semantic model, processing the second feature sequence to obtain an adversarial feature sequence;

[0014] The adversarial feature sequence is converted into adversarial malicious traffic, and the adversarial malicious traffic is sent from the sending end to the receiving end, and feedback information is obtained according to the number of data packets successfully received by the receiving end, so as to combine the feedback information with a preset penalty mechanism to optimize the network of the deep reinforcement learning model to obtain a trained deep reinforcement learning model.

[0015] In addition, the malicious traffic detection and counterattack method according to the above embodiment of the present invention may also have the following additional technical features:

[0016] Furthermore, in one embodiment of the present invention, the first feature sequence includes a packet size feature sequence and a packet interval feature sequence of benign Internet traffic data; the second feature sequence includes a packet size feature sequence and a packet interval feature sequence of preset malicious traffic data; the traffic semantic model is composed of several layers of encoder stacks; wherein each encoder includes a self-attention layer, a bidirectional-cross attention layer and a feedforward neural network layer; the bidirectional-cross attention layer is composed of symmetrical cross attention layers, and different layers are connected by residual connections and layer normalization; the residual connection is used for the input of the traffic semantic model network to skip a certain layer and directly add it to the subsequent layer; the layer normalization is used to perform feature normalization processing within each layer of the network; the deep reinforcement learning model is used for training an environment including a sender of malicious traffic, a target malicious traffic detection system, and a receiver of malicious traffic.

[0017] Furthermore, in one embodiment of the present invention, the deep reinforcement learning model includes a policy network and a Q network; after obtaining the trained deep reinforcement learning model, the method further includes:

[0018] Extracting features of malicious traffic extracted from the anti-malicious traffic to obtain a third feature sequence; wherein the third feature sequence includes a packet size feature sequence and a packet interval feature sequence of the malicious traffic extracted from the anti-malicious traffic;

[0019] Processing the third feature sequence based on the policy network and the trained traffic semantic model to obtain a final adversarial feature sequence;

[0020] The final adversarial feature sequence is converted into the final adversarial malicious traffic.

[0021] Further, in one embodiment of the present invention, extracting the first feature sequence includes:

[0022] Downloading from an open source Internet traffic data set, or capturing traffic data from the Internet, and performing data preprocessing on the traffic data to obtain preprocessed data;

[0023] The preprocessed data is divided into streams, and the packet size feature sequence and the packet interval feature sequence of the preprocessed data are respectively extracted in streams to obtain a first feature sequence; wherein each element in the packet size feature sequence of the preprocessed data is the size of a corresponding data packet, and each element in the packet interval feature sequence of the preprocessed data is the time interval between the transmission of the corresponding data packet and the previous data packet;

[0024] The extracted first feature sequence is padded and divided into blocks.

[0025] Furthermore, in one embodiment of the present invention, padding and blocking the extracted first feature sequence include:

[0026] Determine a first preset length of the first feature sequence, and perform padding processing on feature sequences whose length is less than the first preset length, that is, add a number of zero values ​​at the end of the sequence until the preset length is reached;

[0027] For feature sequences whose length is greater than the first preset length, block processing is performed, that is, starting from each element of the sequence, subsequences with a length of the first preset length are sequentially intercepted as new sequences, until it is impossible to further intercept new sequences that meet the first preset length requirement from the original sequence.

[0028] Furthermore, in one embodiment of the present invention, taking the first feature sequence as an input sequence comprises:

[0029] Inputting the first feature sequence into a self-attention layer to calculate the attention weight of each element in a single input sequence with respect to all other elements, and generating a weighted output sequence based on the attention weight;

[0030] Inputting two equal-length input sequences in the first feature sequence into a symmetrical cross-attention layer based on the weighted output sequence to convert the first input sequence into a first query vector, a first key vector, and a first value vector; and converting the second input sequence into a second query vector, a second key vector, and a second value vector;

[0031] Obtain a first output sequence based on the first query vector, the second key vector, and the second value vector, and obtain a second output sequence based on the second query vector, the first key vector, and the first value vector;

[0032] The first output sequence and the second output sequence are input into the feed-forward neural network layer to obtain the final output sequence.

[0033] Further, in one embodiment of the present invention, a part of the sequence in the input sequence is randomly masked, and the original value of the masked part of the sequence is predicted by using the initialized traffic semantic model to train the traffic semantic model to obtain a trained traffic semantic model, including:

[0034] Randomly masking a preset number of elements in the first feature sequence, and replacing the preset number of elements with mask marks, so as to predict the masked element values ​​through the traffic semantic model;

[0035] The traffic semantic model is optimized according to the prediction results and using forward propagation and back propagation algorithms to obtain a trained traffic semantic model.

[0036] Further, in one embodiment of the present invention, the extracted second feature sequence is processed using a deep reinforcement learning model to obtain action data, and the second feature sequence is processed based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence, including:

[0037] Extracting a packet size feature sequence and a packet interval feature sequence of preset malicious traffic data to obtain a second feature sequence;

[0038] Extracting subsequences of the second feature sequence and connecting them end to end to form a feature sequence of a second preset length to input into the policy network of the deep reinforcement learning model;

[0039] The output action value of the strategy network is a positive integer, and the preset action value is a. If a is an odd number, the value at the packet interval feature sequence (a-1) / 2 position of the feature sequence of the second preset length is set as a mask; if a is an even number, a mask is added between the packet size feature sequence and the packet interval feature sequence a / 2 and a / 2+1 positions of the feature sequence of the second preset length to form a new feature sequence;

[0040] The two new feature sequences are input into the trained traffic semantic model, and the predicted value of the trained traffic semantic model for the mask position replaces the mask to obtain the adversarial feature sequence.

[0041] Furthermore, in one embodiment of the present invention, the countermeasure feature sequence is converted into countermeasure malicious traffic, and the countermeasure malicious traffic is sent from the sending end to the receiving end, and feedback information is obtained according to the number of data packets successfully received by the receiving end, including:

[0042] For the packet size confrontation sequence feature that has not been adjusted in the characteristic sequence of the second preset length, directly restore the original data packet; otherwise, add a new data packet according to the packet size confrontation sequence feature, fill in the correct header information according to other data packets, and fill in the random payload;

[0043] According to the packet interval countermeasure sequence feature, the timestamp of all subsequent data packets is recalculated based on the timestamp of the first data packet;

[0044] Adjust the lifetime field of the header of the newly added data packet so that the newly added data packet is discarded after passing through the malicious traffic detection system and is not received by the receiving end;

[0045] The number of valid malicious data packets successfully received by the receiving end is calculated, and a feedback value is obtained based on the number of valid malicious data packets.

[0046] Furthermore, in one embodiment of the present invention, the method further includes:

[0047] The feedback value r E, distance loss r D and malicious loss M The combination is the sum of the three, namely:

[0048] r=r E +ζr D +ψr M

[0049] Among them, ζ and ψ are hyper parameters, and the distance loss r D is the distance between the feature sequences before and after the above steps of obtaining the counter feature sequence, expressed as edit distance; the malicious loss r M The malicious properties of malicious traffic corresponding to the feature sequences before and after the steps of obtaining the counter feature sequences are compared.

[0050] To achieve the above object, another object of the present invention is to provide a malicious traffic detection and anti-attack device, comprising:

[0051] A semantic model initialization module, used for extracting a first feature sequence, and using the first feature sequence as an input sequence of an initialized traffic semantic model to perform semantic model training;

[0052] A semantic model training module is used to randomly mask a part of the sequence in the input sequence, and use the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model;

[0053] An adversarial feature sequence generation module, used to process the extracted second feature sequence using a deep reinforcement learning model to obtain action data, and to process the second feature sequence based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence;

[0054] A reinforcement learning model training module is used to convert the adversarial feature sequence into adversarial malicious traffic, and send the adversarial malicious traffic from the sending end to the receiving end, and obtain feedback information according to the number of data packets successfully received by the receiving end, so as to combine the feedback information with a preset penalty mechanism, optimize the network of the deep reinforcement learning model, and obtain a trained deep reinforcement learning model;

[0055] The model inference module is used to extract features of malicious traffic extracted from the anti-malicious traffic to obtain a third feature sequence; wherein the third feature sequence includes a packet size feature sequence and a packet interval feature sequence of the malicious traffic extracted from the anti-malicious traffic, and the third feature sequence is processed based on the policy network and the trained traffic semantic model to obtain a final anti-malicious feature sequence, so as to convert the final anti-malicious feature sequence into a final anti-malicious traffic.

[0056] The malicious traffic detection counterattack method and device of the embodiment of the present invention involve two stages: benign traffic cloning and counterattack traffic generation. In the benign traffic cloning stage, a traffic semantic model designed for network traffic is trained and deployed to learn the sending mode of massive benign traffic in the Internet; in the counterattack traffic generation stage, based on the feedback of the target detection system, combined with deep reinforcement learning and traffic semantic model, the local sending mode of benign traffic is embedded in the malicious traffic to obtain counterattack malicious traffic, and a specific penalty mechanism is introduced to ensure the original traffic properties of the generated traffic.

[0057] The beneficial effects of the present invention are:

[0058] The black-box adversarial attack scheme based on traffic semantic model and deep reinforcement learning proposed in this paper can effectively generate practical adversarial malicious traffic, thus providing a powerful tool for verifying and improving the robustness of malicious traffic detection systems. At the same time, by simulating adversarial attack scenarios in the real world, it can help researchers identify potential weaknesses in existing detection systems, and then design more effective defense systems and technical solutions to improve network security protection levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0060] Figure 1 Flow chart of a method for malicious traffic detection and counterattack according to an embodiment of the present invention;

[0061] Figure 2 A logical schematic diagram of a malicious traffic detection and counterattack method according to an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of the principle of a method for detecting malicious traffic and countering attacks according to an embodiment of the present invention;

[0063] Figure 4 Schematic diagram of the structure of a malicious traffic detection and anti-attack device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0065] The following describes a malicious traffic detection and counter-attack method and device according to an embodiment of the present invention with reference to the accompanying drawings.

[0066] The embodiment of the present invention is directed to a black box counterattack against a malicious traffic detection system that is based on a decision tree and performs detection in units of flows. The malicious traffic is malware traffic, and its maliciousness is reflected in the load.

[0067] The present invention combines traffic semantic models with deep reinforcement learning technology, and designs a black box adversarial attack technology for malicious traffic detection systems based on machine learning architecture. This allows the present invention to generate adversarial malicious traffic that can escape detection and maintain its original traffic properties without knowing the specific parameters and implementation details of the target detection system. The present invention as a whole can be divided into two stages: a benign traffic cloning stage and an adversarial traffic generation stage. In the benign traffic cloning stage, a traffic semantic model Traffic-BERT suitable for the network field is designed based on the BERT model, and massive benign traffic data in the Internet is collected to learn the sending mode of benign traffic; in the adversarial traffic generation stage, based on the policy network, Q network and Traffic-BERT, it is determined how to effectively embed benign traffic patterns into malicious traffic, and the generation strategy is optimized by integrating the feedback of the target detection system and a specific penalty mechanism to ensure that the adversarial malicious traffic is difficult to detect and maintains the necessary original traffic properties.

[0068] Figure 1 The present invention is a flowchart of a method for detecting malicious traffic and countering attacks according to an embodiment of the present invention.

[0069] like Figure 1 , Figure 2 and Figure 3 As shown, the method includes but is not limited to the following steps:

[0070] S1, extracting a first feature sequence, and using the first feature sequence as an input sequence of an initialized traffic semantic model.

[0071] In this stage, benign traffic cloning is performed. In this stage, benign Internet traffic is collected as training data and data processing is performed. The data processing method includes extracting packet feature sequences and packet interval feature sequences, and filling and blocking the extracted feature sequences.

[0072] The first feature sequence extracted in this embodiment is extracted based on the collected benign Internet traffic. The method of collecting benign Internet traffic is to download data from any open source Internet traffic data set, or to capture traffic data from the Internet, and then perform data preprocessing, wherein the data processing is to remove obviously abnormal data therein.

[0073] The way to remove obviously abnormal data is to identify and exclude data with obvious malicious traffic characteristics. For example, filter out traffic from the same IP address scanning a large number of random addresses, traffic with an abnormally high sending rate that may be a DoS attack, and other traffic with specific malicious characteristics.

[0074] Among them, the first feature sequence is extracted, that is, the packet size feature sequence and packet interval feature sequence of the benign Internet traffic data are extracted. The packet size feature sequence and packet interval feature sequence are divided into all data packets in units of flows. The definition of a flow can be a five-tuple of <source IP address, destination IP address, source port, destination port, protocol>, or other reasonable definitions. The packet size feature sequence is extracted in units of flows, and each element in the sequence is the size of the corresponding data packet; the packet interval feature sequence is extracted in units of flows, and each element in the sequence is the time interval between the corresponding data packet and the previous data packet, and the first element is set to 0.

[0075] Furthermore, the method of padding and blocking the extracted packet size feature sequence and packet interval feature sequence is to set the fixed length of the feature sequence to n=512, and perform padding processing on the feature sequence with a length less than n, that is, add a number of zero values ​​at the end of the sequence until the fixed length n is reached; and perform blocking processing on the feature sequence with a length greater than n, that is, starting from each element of the sequence, sequentially intercept subsequences of length n as new sequences until it is impossible to continue to intercept new sequences that meet the fixed length requirements from the original sequence.

[0076] Furthermore, this step initializes the traffic semantic model Traffic-BERT, including initializing the model structure and parameters.

[0077] The Traffic-BERT model of the embodiment of the present invention is composed of several layers of encoders stacked together. Each encoder includes a self-attention layer, a bidirectional-cross attention layer and a feedforward neural network layer, and different layers are tightly connected through residual connections and layer normalization. Traffic-BERT uses packet size feature sequences and packet interval feature sequences as model inputs, and performs data processing through a self-attention layer, a bidirectional-cross attention layer and a feedforward neural network layer.

[0078] Among them, the first feature sequence is input into the self-attention layer. For a given single input sequence, the self-attention layer calculates the attention weight of each element in the sequence for all other elements, and generates a weighted output sequence based on these weights. Its calculation process can be described as follows: each element of the input sequence is converted into three vectors - query vector Q, key vector K and value vector V. These three vectors are obtained by linear mapping with the input elements; based on these three vectors, the output of the self-attention layer is:

[0079]

[0080] Among them, d k is the dimension of the key vector.

[0081] Among them, the two equal-length input sequences in the first feature sequence are input into the symmetrical cross-attention layer; the bidirectional-cross-attention layer is composed of a symmetrical cross-attention layer, receiving two equal-length sequences as input, and its calculation process can be expressed as: the first input sequence is converted into a query vector Q 1 , key vector K 1 Sum value vector V 1 , the second input sequence is converted into a query vector Q 2 , key vector K 2 Sum value vector V 2 , the output of the bidirectional-cross attention layer corresponding to the first input sequence is Attn(Q 1 ,K 2 ,V 2 ), the output of the bidirectional-cross attention layer corresponding to the second input sequence is Attn(Q 2 ,K 1 ,V 1 ).

[0082] Further, the first output sequence Attn(Q 1 ,K 2 ,V 2 ) and the second output sequence Attn(Q 2 ,K 1 ,V 1 ) is input into the feed-forward neural network layer to obtain the final output sequence.

[0083] It can be understood that residual connection means that the input of the network skips a certain layer and is directly added to the subsequent layer; layer normalization is used to perform feature normalization processing within each layer of the network, which can be expressed as:

[0084]

[0085] Among them, μ and σ are the mean and standard deviation of all neurons in this layer, and γ and β are learnable parameters.

[0086] Furthermore, Traffic-BERT is initialized by random normal distribution, thereby completing the initialization of the Traffic-BERT model.

[0087] S2, randomly masking part of the sequence in the input sequence, and using the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model.

[0088] It can be understood that this step trains Traffic-BERT. After Traffic-BERT training is completed, the parameters of Traffic-BERT are fixed to complete benign traffic cloning.

[0089] It is understandable that the training task of the Traffic-BERT model is a masked language model task, that is, by randomly masking a part of the input sequence, and then letting the model predict the original value of these masked parts. Specifically, a preset number of elements in the packet size feature sequence and the packet interval feature sequence, which can be p%, are randomly masked and replaced with a special [MASK] tag, and then Traffic-BERT is allowed to predict the value of the masked elements.

[0090] Furthermore, based on the above prediction results, Traffic-BERT is optimized using forward propagation and back propagation algorithms. The optimizer used in the training may be an Adam optimizer, and the loss function may be a cross entropy loss function.

[0091] In one embodiment of the present invention, the method for judging whether Traffic-BERT training is completed is that the number of training rounds exceeds the target number of rounds Ep, or the model performance no longer improves within the number of rounds El.

[0092] S3, using the deep reinforcement learning model to process the extracted second feature sequence to obtain action data, and based on the action data and the trained traffic semantic model, processing the second feature sequence to obtain an adversarial feature sequence.

[0093] It can be seen that this stage is the generation of adversarial traffic. This stage mainly establishes the deep reinforcement learning framework, initializes the policy network and Q network, and starts the training of the deep reinforcement learning model.

[0094] It can be understood that establishing a deep reinforcement learning framework means establishing a deep reinforcement learning framework with the Soft Actor-Critic (SAC) algorithm as the main body, while learning a policy network and a Q network to maximize the long-term reward and entropy of the policy.

[0095] The main body of the policy network and Q network of the deep reinforcement learning model of the embodiment of the present invention is a Gated Recurrent Unit (GRU) network, which takes the concatenation of the packet size feature sequence and the packet interval feature sequence as input. The output of the policy network is a discrete value used to indicate the action at time step t; the output of the Q network is a vector used to evaluate the state-action value at time step t.

[0096] Among them, the policy network and Q network are initialized by normal distribution random initialization.

[0097] If the deep reinforcement learning training environment is not initialized, initialize it; otherwise, reset the environment to its initial state.

[0098] The deep reinforcement learning training environment includes the sender of malicious traffic, the target malicious traffic detection system and the receiver of malicious traffic. The target detection system is a deployed detection system, and its internal implementation details cannot be observed; the sender and the receiver are located upstream and downstream of the target detection system respectively, and malicious traffic can be judged by the traffic arriving at the receiver. Initializing the environment means setting the sender and the receiver to normal working state and extracting a malicious flow from the pre-prepared malicious traffic data.

[0099] Furthermore, the present invention performs feature processing on malicious flows, extracts a second feature sequence, namely, a packet size feature sequence and a packet interval feature sequence of malicious traffic data, and performs length processing. The policy network receives the feature sequence and takes action. The feature sequence is processed in combination with the action and Traffic-BERT to obtain an adversarial feature sequence.

[0100] Exemplarily, the manner of extracting the second feature sequence, the packet size feature sequence and the packet interval feature sequence is consistent with that described above and will not be described in detail herein.

[0101] Exemplarily, the second characteristic sequence is processed in length by intercepting a subsequence of the first n elements of the sequence. If the sequence length is less than n, the original sequence is padded according to the same steps as above, and no redundant description is given here.

[0102] Exemplarily, a subsequence of the second feature sequence is intercepted and connected end to end to form a feature sequence of a second preset length to be input into the policy network of the deep reinforcement learning model; specifically, the policy network receives the feature sequence by concatenating a packet size feature sequence and a packet interval feature sequence with a length of n end to end to form a feature sequence with a length of 2n.

[0103] Exemplarily, the output action of the policy network is a positive integer, and the value range is [0, 2n].

[0104] The embodiment of the present invention first adjusts the malicious feature sequence according to the action value output by the policy network: assuming the action is a, if a is an odd number, the value at the position (a-1) / 2 of the packet interval feature sequence is set to [MASK], and the packet size sequence remains unchanged; if a is an even number, a [MASK] is added between the positions a / 2 and a / 2+1 of the packet size sequence and the packet interval sequence to form a new feature sequence. The two new feature sequences are then sent to Traffic-BERT, and the predicted value of Traffic-BERT for the [MASK] position is replaced by [MASK], thereby obtaining an adversarial feature sequence.

[0105] S4, converting the adversarial feature sequence into adversarial malicious traffic, and sending the adversarial malicious traffic from the sending end to the receiving end, obtaining feedback information according to the number of data packets successfully received by the receiving end, combining the feedback information with a preset penalty mechanism, optimizing the network of the deep reinforcement learning model to obtain a trained deep reinforcement learning model.

[0106] The embodiment of the present invention converts the confrontation feature sequence into confrontation malicious traffic, sends it from the sending end to the receiving end, obtains feedback according to the number of data packets successfully received by the receiving end, and combines the feedback with a specific penalty mechanism to optimize the policy network and the Q network.

[0107] Specifically, the method of converting the countermeasure feature sequence into the countermeasure malicious traffic is to directly restore the original data packet for the packet size sequence feature that has not been adjusted after step S4; otherwise, the feature is a newly added feature, and a new data packet should be added according to the packet size countermeasure sequence feature, and the correct header information should be filled in according to other data packets, and the random payload should be filled to ensure that the size of the newly added data packet is equal to the newly added feature. According to the packet interval countermeasure sequence feature, the timestamp of all subsequent data packets is recalculated based on the timestamp of the first data packet. The lifetime field of the header of the newly added data packet is adjusted so that it can pass the malicious traffic detection system, but will not be received by the receiving end, thereby ensuring that the payload information of the original malicious flow remains unchanged.

[0108] The method for obtaining feedback based on the number of data packets successfully received by the receiving end is to calculate the number of valid malicious data packets successfully received by the receiving end, and divide the number by n as the feedback value r. E .

[0109] Among them, the specific penalty mechanism includes the distance loss r D Malicious loss M The way to combine feedback with a specific punishment mechanism is to add the three together, namely:

[0110] r=r E +ζr D +ψr M

[0111] Among them, ζ=0.1 and ψ=0.1 are hyper parameters.

[0112] Among them, the distance loss r D It is used to measure the distance between the feature sequences before and after the above steps to obtain the adversarial feature sequence, expressed as edit distance. Since the original feature sequence and the adversarial feature sequence at any time step have one and only one difference, so:

[0113] r D =-1

[0114] Among them, malicious loss rM It is used to measure the degree to which the feature sequence before and after the above steps to obtain the counter feature sequence destroys the maliciousness of the original malicious traffic. For example, for DoS traffic, r M It can be defined as the difference between the sending rates of the previous and next feature sequences.

[0115] The method for determining whether the trajectory of fighting against malicious traffic has ended is to determine whether the feedback value r E Is it greater than the threshold r? ε = 0.99. If yes, the trajectory ends, otherwise it does not end.

[0116] If it has not ended, take the currently modified adversarial malicious flow as the new malicious flow, and return to step S4 to perform feature processing on the malicious flow to obtain the adversarial feature sequence; otherwise, determine whether the reinforcement learning model training is finished. Among them, the method for determining whether the reinforcement learning model training is finished can be to determine whether the number of training trajectories exceeds the target number of trajectories Epi=2000. If it has ended, proceed to the next step. Otherwise, return to step S4 to initialize the environment if it is not initialized; otherwise, reset the environment to the initial state.

[0117] Furthermore, feature extraction is performed on malicious traffic extracted from the anti-malicious traffic to obtain a third feature sequence; wherein the third feature sequence includes a packet size feature sequence and a packet interval feature sequence of the malicious traffic extracted from the anti-malicious traffic; the third feature sequence is processed based on the policy network and the trained traffic semantic model to obtain a final anti-malicious feature sequence; and the final anti-malicious feature sequence is converted into a final anti-malicious traffic.

[0118] Specifically, after the training of the above model is completed, the adversarial malicious traffic inference begins. Extract malicious flows from malicious traffic data, extract the third feature sequence, packet size feature sequence and packet interval feature sequence, and perform length processing. Repeatedly combine the policy network and Traffic-BERT to process the feature sequence until the generation process is completed, and finally obtain the adversarial feature sequence. The adversarial feature sequence is converted into adversarial malicious traffic, and the adversarial traffic generation phase ends.

[0119] The method of extracting the packet size feature sequence and the packet interval feature sequence is consistent with the above steps and will not be described in detail here.

[0120] The length processing method is the same as the above steps and will not be described in detail here.

[0121] The method of combining the strategy network with Traffic-BERT to process the feature sequence is consistent with the above steps and will not be elaborated here.

[0122] The method for judging the end of the generation process is to determine the end of the generation process according to the action a at time step t. t , corresponding to the Q network output Is it greater than the target threshold Q? T , or the time step t is greater than a target threshold τ, if so, the generation process ends.

[0123] The malicious traffic detection counterattack method of the embodiment of the present invention first trains and deploys a traffic semantic model designed for network traffic to learn the sending pattern of massive benign traffic on the Internet, and then introduces a specific penalty mechanism to ensure the original traffic properties of the generated traffic. Deep reinforcement learning technology is used to process the feedback of the target detection system on malicious traffic, and combined with the traffic semantic model, counterattack malicious traffic that can escape detection is generated.

[0124] Furthermore, if Figure 4 As shown, in order to implement the above embodiment, the present invention proposes a malicious traffic detection and anti-attack device 10, the device 10 includes:

[0125] The semantic model initialization module 100 is used to randomly mask a part of the sequence in the input sequence, and use the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model;

[0126] The semantic model training module 200 is used to randomly mask a part of the sequence in the input sequence, and use the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model;

[0127] An adversarial feature sequence generation module 300, configured to process the extracted second feature sequence using a deep reinforcement learning model to obtain action data, and to process the second feature sequence based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence;

[0128] A reinforcement learning model training module 400 is used to convert the adversarial feature sequence into adversarial malicious traffic, and send the adversarial malicious traffic from the sending end to the receiving end, and obtain feedback information according to the number of data packets successfully received by the receiving end, so as to combine the feedback information with a preset penalty mechanism to optimize the network of the deep reinforcement learning model to obtain a trained deep reinforcement learning model;

[0129] The model inference module 500 is used to extract features from malicious traffic extracted from the anti-malicious traffic to obtain a third feature sequence; wherein the third feature sequence includes a packet size feature sequence and a packet interval feature sequence of the malicious traffic extracted from the anti-malicious traffic; the third feature sequence is processed based on the policy network and the trained traffic semantic model to obtain a final anti-malicious feature sequence; and the final anti-malicious feature sequence is converted into a final anti-malicious traffic.

[0130] The malicious traffic detection and anti-attack device of the embodiment of the present invention first trains and deploys a traffic semantic model designed for network traffic to learn the sending mode of massive benign traffic on the Internet, and then introduces a specific penalty mechanism to ensure the original traffic properties of the generated traffic. The deep reinforcement learning technology is used to process the feedback of the target detection system on the malicious traffic, and combined with the traffic semantic model, it generates anti-malicious traffic that can escape detection.

[0131] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0132] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0133] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for detecting malicious traffic and countering attacks, characterized in that: include: Extracting a first feature sequence, and using the first feature sequence as an input sequence of an initialized traffic semantic model; Randomly masking part of the sequence in the input sequence, and using the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model; Using a deep reinforcement learning model to process the extracted second feature sequence to obtain action data, and based on the action data and the trained traffic semantic model, processing the second feature sequence to obtain an adversarial feature sequence; The adversarial feature sequence is converted into adversarial malicious traffic, and the adversarial malicious traffic is sent from the sending end to the receiving end, and feedback information is obtained according to the number of data packets successfully received by the receiving end, so as to combine the feedback information with a preset penalty mechanism to optimize the network of the deep reinforcement learning model to obtain a trained deep reinforcement learning model.

2. The method according to claim 1, characterized in that The first feature sequence includes a packet size feature sequence and a packet interval feature sequence of benign Internet traffic data; the second feature sequence includes a packet size feature sequence and a packet interval feature sequence of preset malicious traffic data; the traffic semantic model is composed of a plurality of layers of encoder stacks; wherein each encoder comprises a self-attention layer, a bidirectional-cross attention layer and a feedforward neural network layer; the bidirectional-cross attention layer is composed of symmetrical cross attention layers, and different layers are connected by residual connections and layer normalization; the residual connection is used for the input of the traffic semantic model network to skip a certain layer and directly add it to the subsequent layer; the layer normalization is used to perform feature normalization processing within each layer of the network; the deep reinforcement learning model is used for training an environment including a sender of malicious traffic, a target malicious traffic detection system, and a receiver of malicious traffic.

3. The method according to claim 2, characterized in that The deep reinforcement learning model includes a policy network and a Q network; After obtaining the trained deep reinforcement learning model, the method further includes: Extracting features of malicious traffic extracted from the anti-malicious traffic to obtain a third feature sequence; wherein the third feature sequence includes a packet size feature sequence and a packet interval feature sequence of the malicious traffic extracted from the anti-malicious traffic; Processing the third feature sequence based on the policy network and the trained traffic semantic model to obtain a final adversarial feature sequence; The final adversarial feature sequence is converted into the final adversarial malicious traffic.

4. The method according to claim 1, characterized in that: Extracting the first feature sequence includes: Downloading from an open source Internet traffic data set, or capturing traffic data from the Internet, and performing data preprocessing on the traffic data to obtain preprocessed data; The preprocessed data is divided into streams, and the packet size feature sequence and the packet interval feature sequence of the preprocessed data are respectively extracted in streams to obtain a first feature sequence; wherein each element in the packet size feature sequence of the preprocessed data is the size of a corresponding data packet, and each element in the packet interval feature sequence of the preprocessed data is the time interval between the transmission of the corresponding data packet and the previous data packet; The extracted first feature sequence is padded and divided into blocks.

5. The method according to claim 4, characterized in that The extracted first feature sequence is filled and divided into blocks, including: Determine a first preset length of the first feature sequence, and perform padding processing on feature sequences whose length is less than the first preset length, that is, add a number of zero values ​​at the end of the sequence until the preset length is reached; For feature sequences whose length is greater than the first preset length, block processing is performed, that is, starting from each element of the sequence, subsequences with a length of the first preset length are sequentially intercepted as new sequences, until it is impossible to further intercept new sequences that meet the first preset length requirement from the original sequence.

6. The method according to claim 1, characterized in that Taking the first feature sequence as an input sequence, comprising: Inputting the first feature sequence into a self-attention layer to calculate the attention weight of each element in a single input sequence with respect to all other elements, and generating a weighted output sequence based on the attention weight; Inputting two equal-length input sequences in the first feature sequence into a symmetrical cross-attention layer based on the weighted output sequence to convert the first input sequence into a first query vector, a first key vector, and a first value vector; and converting the second input sequence into a second query vector, a second key vector, and a second value vector; Obtain a first output sequence based on the first query vector, the second key vector, and the second value vector, and obtain a second output sequence based on the second query vector, the first key vector, and the first value vector; The first output sequence and the second output sequence are input into the feed-forward neural network layer to obtain the final output sequence.

7. The method according to claim 1, characterized in that Randomly masking a part of the sequence in the input sequence, and using the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model, including: Randomly masking a preset number of elements in the first feature sequence, and replacing the preset number of elements with mask marks, so as to predict the masked element values ​​through the traffic semantic model; The traffic semantic model is optimized according to the prediction results and using forward propagation and back propagation algorithms to obtain a trained traffic semantic model.

8. The method according to claim 2, characterized in that: The extracted second feature sequence is processed by using a deep reinforcement learning model to obtain action data, and the second feature sequence is processed based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence, including: Extracting a packet size feature sequence and a packet interval feature sequence of preset malicious traffic data to obtain a second feature sequence; Extracting subsequences of the second feature sequence and connecting them end to end to form a feature sequence of a second preset length to input into the policy network of the deep reinforcement learning model; The output action value of the strategy network is a positive integer, and the preset action value is a. If a is an odd number, the value at the packet interval feature sequence (a-1) / 2 position of the feature sequence of the second preset length is set as a mask; if a is an even number, a mask is added between the packet size feature sequence and the packet interval feature sequence a / 2 and a / 2+1 positions of the feature sequence of the second preset length to form a new feature sequence; The two new feature sequences are input into the trained traffic semantic model, and the predicted value of the trained traffic semantic model for the mask position replaces the mask to obtain the adversarial feature sequence.

9. The method according to claim 8, characterized in that The countermeasure feature sequence is converted into countermeasure malicious traffic, and the countermeasure malicious traffic is sent from the sending end to the receiving end, and feedback information is obtained according to the number of data packets successfully received by the receiving end, including: For the packet size confrontation sequence feature that has not been adjusted in the characteristic sequence of the second preset length, directly restore the original data packet; otherwise, add a new data packet according to the packet size confrontation sequence feature, fill in the correct header information according to other data packets, and fill in the random payload; According to the packet interval countermeasure sequence feature, the timestamp of all subsequent data packets is recalculated based on the timestamp of the first data packet; Adjust the lifetime field of the header of the newly added data packet so that the newly added data packet is discarded after passing through the malicious traffic detection system and is not received by the receiving end; The number of valid malicious data packets successfully received by the receiving end is calculated, and a feedback value is obtained based on the number of valid malicious data packets.

10. The method according to claim 9, characterized in that The method further comprises: The feedback value r E , distance loss r D and malicious loss M The combination is the sum of the three, namely: r=r E +ζr D +ψr M Among them, ζ and ψ are hyper parameters, and the distance loss r D is the distance between the feature sequences before and after the above steps of obtaining the counter feature sequence, expressed as edit distance; the malicious loss r M The malicious properties of malicious traffic corresponding to the feature sequences before and after the steps of obtaining the counter feature sequences are compared.

11. A malicious traffic detection and anti-attack device, characterized in that: include: A semantic model initialization module, used for extracting a first feature sequence, and using the first feature sequence as an input sequence of an initialized traffic semantic model to perform semantic model training; A semantic model training module is used to randomly mask a part of the sequence in the input sequence, and use the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model; An adversarial feature sequence generation module, used to process the extracted second feature sequence using a deep reinforcement learning model to obtain action data, and to process the second feature sequence based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence; A reinforcement learning model training module is used to convert the adversarial feature sequence into adversarial malicious traffic, and send the adversarial malicious traffic from the sending end to the receiving end, and obtain feedback information according to the number of data packets successfully received by the receiving end, so as to combine the feedback information with a preset penalty mechanism, optimize the network of the deep reinforcement learning model, and obtain a trained deep reinforcement learning model; The model inference module is used to extract features of malicious traffic extracted from the anti-malicious traffic to obtain a third feature sequence; wherein the third feature sequence includes a packet size feature sequence and a packet interval feature sequence of the malicious traffic extracted from the anti-malicious traffic, and the third feature sequence is processed based on the policy network and the trained traffic semantic model to obtain a final anti-malicious feature sequence, so as to convert the final anti-malicious feature sequence into a final anti-malicious traffic.

Citation Information

Patent Citations

  • Intrusion detection and semantic decision tree quantitative interpretation method based on convolutional neural network

    CN112491796A

  • Malicious domain name detection method and device based on deep reinforcement learning

    CN114050912A

  • Malicious traffic protection method and system based on active detection

    CN116015788A

  • Android malicious software escape attack enhancement detection method based on anti-purification strategy

    CN117494127A

  • Intensive care data remote submission sorting method and system based on cloud computing

    CN118366597A