A method and device for detecting malicious traffic and countering attacks

Abstract

The present invention discloses a method and device for detecting and countering malicious traffic attacks. The method comprises: using a first feature sequence as an input sequence of an initialized traffic semantic model; randomly masking a portion of the input sequence and predicting the original value of the masked portion to train the traffic semantic model; processing the extracted second feature sequence using a deep reinforcement learning model to obtain action data, and processing the second feature sequence based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence; converting the adversarial feature sequence into adversarial malicious traffic and sending it from a transmitting end to a receiving end, obtaining feedback information based on the number of data packets successfully received by the receiving end, combining the feedback information with a preset penalty mechanism, and optimizing the network of the deep reinforcement learning model to obtain a trained deep reinforcement learning model. The present invention can generate adversarial malicious traffic that can evade detection while maintaining its original traffic properties under unknown circumstances.

Description

Technical Field

[0001] The present invention relates to the field of cyberspace security and artificial intelligence security technology, and in particular to a method and device for detecting malicious traffic and countering attacks. Background Art

[0002] Malicious traffic detection technology is a key research area in network security, primarily categorized into signature-based and machine learning-based malicious traffic detection systems. Current signature-based systems rely on predefined malicious signature libraries for traffic analysis. While effective in detecting known threats, this approach is limited when faced with new traffic attack patterns, such as zero-day attacks. In contrast, machine learning-based detection systems analyze traffic behavior patterns and leverage machine learning techniques to automatically identify unknown or mutated malicious traffic, demonstrating greater flexibility and adaptability.

[0003] Adversarial attacks, on the other hand, are an attack technique that interacts with machine learning models and applies subtle perturbations to the input data to confuse the model's output. Currently, adversarial attacks can be categorized into two types: white-box and black-box. White-box attacks assume the attacker has complete knowledge of the target model's internal structure and parameters, while black-box attacks operate with only knowledge of the model's inputs and outputs. Black-box adversarial attacks are therefore more practical and important in cybersecurity research because they more closely resemble real-world attack scenarios.

[0004] Generative models are a type of machine learning model that automatically generates data by learning from the distribution of large amounts of data. In the field of natural language processing, BERT (Bidirectional Encoder Representations from Transformers) is an advanced generative model that captures the deep semantic relationships in sequence data through a specific multi-layer bidirectional encoder structure.

[0005] Furthermore, deep reinforcement learning has been applied to a variety of fields, including cyberspace security, next-generation internet, robotics, autonomous driving, and healthcare. Generally, reinforcement learning methods aim to solve the problem of how an intelligent agent interacts with its environment to maximize cumulative rewards. These problems can typically be modeled as a Markov decision process (MDP) = (S, A, P, R, T). Here, S represents the agent's state space, i.e., the set of possible states the agent can be in the environment; A represents the agent's action space, i.e., the set of possible actions the agent can take; P represents the state transition function, which describes the probability distribution of the next state the agent can reach after executing an action; R represents the reward function, i.e., the benefit the agent obtains from the environment after taking an action; and T represents the timeframe of the decision-making process, which describes the termination condition of the decision. Deep reinforcement learning combines deep learning techniques with reinforcement learning, with the agent making decisions using neural networks.

[0006] There is an urgent need for a method to implement adversarial attacks against malicious traffic detection systems in black box scenarios in order to evaluate the robustness of malicious traffic detection systems. Summary of the Invention

[0007] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0008] To this end, the present invention proposes a malicious traffic detection counter-attack method, which aims to generate counter-malicious traffic that can evade detection while maintaining its original traffic properties without knowing the specific parameters and implementation details of the target detection system.

[0009] Another object of the present invention is to provide a malicious traffic detection and counter-attack device.

[0010] To achieve the above objectives, an embodiment of the present invention provides a method for detecting malicious traffic and resisting attacks, including:

[0011] Extracting a first feature sequence, and using the first feature sequence as an input sequence of an initialized traffic semantic model;

[0012] Randomly masking part of the sequence in the input sequence, and using the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model;

[0013] Using a deep reinforcement learning model to process the extracted second feature sequence to obtain action data, and processing the second feature sequence based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence;

[0014] The adversarial feature sequence is converted into adversarial malicious traffic, and the adversarial malicious traffic is sent from the sending end to the receiving end. Feedback information is obtained based on the number of data packets successfully received by the receiving end, so as to combine the feedback information with a preset penalty mechanism to optimize the network of the deep reinforcement learning model to obtain a trained deep reinforcement learning model.

[0015] In addition, the malicious traffic detection and counterattack method according to the above embodiment of the present invention may also have the following additional technical features:

[0016] Furthermore, in one embodiment of the present invention, the first feature sequence includes a packet size feature sequence and a packet interval feature sequence of benign Internet traffic data; the second feature sequence includes a packet size feature sequence and a packet interval feature sequence of preset malicious traffic data; the traffic semantic model is composed of several layers of encoder stacks; wherein each encoder includes a self-attention layer, a bidirectional-cross attention layer and a feedforward neural network layer; the bidirectional-cross attention layer is composed of symmetrical cross attention layers, and different layers are connected by residual connections and layer normalization; the residual connection is used for the input of the traffic semantic model network to skip a certain layer and directly add it to the subsequent layer; the layer normalization is used to perform feature normalization processing within each layer of the network; the deep reinforcement learning model is used for training an environment including a sender of malicious traffic, a target malicious traffic detection system, and a receiver of malicious traffic.

[0017] Furthermore, in one embodiment of the present invention, the deep reinforcement learning model includes a policy network and a Q network; after obtaining the trained deep reinforcement learning model, the method further includes:

[0018] Extracting features from the malicious traffic extracted from the anti-malicious traffic to obtain a third feature sequence; wherein the third feature sequence includes a packet size feature sequence and a packet interval feature sequence of the malicious traffic extracted from the anti-malicious traffic;

[0019] Processing the third feature sequence based on the policy network and the trained traffic semantic model to obtain a final adversarial feature sequence;

[0020] The final adversarial feature sequence is converted into the final adversarial malicious traffic.

[0021] Furthermore, in one embodiment of the present invention, extracting the first feature sequence includes:

[0022] Downloading traffic data from an open source Internet traffic dataset, or capturing traffic data from the Internet, and performing data preprocessing on the traffic data to obtain preprocessed data;

[0023] The preprocessed data is divided into stream units, and a packet size feature sequence and a packet interval feature sequence of the preprocessed data are extracted in units of streams to obtain a first feature sequence; wherein each element in the packet size feature sequence of the preprocessed data is the size of a corresponding data packet, and each element in the packet interval feature sequence of the preprocessed data is the time interval between the transmission of the corresponding data packet and the previous data packet;

[0024] The extracted first feature sequence is padded and divided into blocks.

[0025] Furthermore, in one embodiment of the present invention, padding and blocking the extracted first feature sequence include:

[0026] Determining a first preset length of the first feature sequence, and performing padding processing on feature sequences whose length is less than the first preset length, that is, adding a number of zero values ​​at the end of the sequence until the preset length is reached;

[0027] For feature sequences whose length is greater than the first preset length, block processing is performed, that is, starting from each element of the sequence, subsequences with the first preset length are sequentially intercepted as new sequences until no new sequences that meet the first preset length requirements can be further intercepted from the original sequence.

[0028] Furthermore, in one embodiment of the present invention, taking the first feature sequence as an input sequence includes:

[0029] Inputting the first feature sequence into a self-attention layer to calculate the attention weight of each element in a single input sequence with respect to all other elements, and generating a weighted output sequence based on the attention weights;

[0030] Inputting two equal-length input sequences in the first feature sequence into a symmetrical cross-attention layer based on the weighted output sequence to convert the first input sequence into a first query vector, a first key vector, and a first value vector; and converting the second input sequence into a second query vector, a second key vector, and a second value vector;

[0031] Obtain a first output sequence based on the first query vector, the second key vector, and the second value vector, and obtain a second output sequence based on the second query vector, the first key vector, and the first value vector;

[0032] The first output sequence and the second output sequence are input into the feed-forward neural network layer to obtain the final output sequence.

[0033] Furthermore, in one embodiment of the present invention, a portion of the input sequence is randomly masked, and the initialized traffic semantic model is used to predict the original value of the masked portion of the sequence to train the traffic semantic model to obtain a trained traffic semantic model, including:

[0034] Randomly masking a preset number of elements in the first feature sequence and replacing the preset number of elements with mask marks to predict the masked element values ​​through the traffic semantic model;

[0035] According to the prediction results, the traffic semantic model is optimized using forward propagation and back propagation algorithms to obtain a trained traffic semantic model.

[0036] Furthermore, in one embodiment of the present invention, the extracted second feature sequence is processed using a deep reinforcement learning model to obtain action data, and the second feature sequence is processed based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence, including:

[0037] Extracting a packet size feature sequence and a packet interval feature sequence of preset malicious traffic data to obtain a second feature sequence;

[0038] Extracting subsequences of the second feature sequence and concatenating them end to end to form a feature sequence of a second preset length for input into the policy network of the deep reinforcement learning model;

[0039] The output action value of the policy network is a positive integer, and the preset action value is a. If a is an odd number, the value at the packet interval feature sequence (a-1) / 2 position of the feature sequence of the second preset length is set as a mask; if a is an even number, a mask is added between the packet size feature sequence and the packet interval feature sequence a / 2 and a / 2+1 positions of the feature sequence of the second preset length to form a new feature sequence;

[0040] The two new feature sequences are input into the trained traffic semantic model, and the predicted value of the trained traffic semantic model for the mask position replaces the mask to obtain the adversarial feature sequence.

[0041] Furthermore, in one embodiment of the present invention, the countermeasure feature sequence is converted into countermeasure malicious traffic, and the countermeasure malicious traffic is sent from the sending end to the receiving end, and feedback information is obtained based on the number of data packets successfully received by the receiving end, including:

[0042] For packet size adversarial sequence features that are not adjusted in the characteristic sequence of the second preset length, directly restore the original data packet; otherwise, add a new data packet based on the packet size adversarial sequence features, fill in the correct packet header information based on other data packets, and fill in the random payload;

[0043] Based on the packet interval countermeasure sequence feature, the timestamp of all subsequent data packets is recalculated based on the timestamp of the first data packet;

[0044] Adjust the Time to Live field in the header of newly added data packets so that they are discarded after passing through the malicious traffic detection system and are not received by the receiving end;

[0045] The number of valid malicious data packets successfully received by the receiving end is calculated, and a feedback value is obtained based on the number of valid malicious data packets.

[0046] Furthermore, in one embodiment of the present invention, the method further includes:

[0047] The feedback value r E, distance loss r D and malicious loss M The combination is the sum of the three, namely:

[0048] r=r E +ζr D +ψr M

[0049] Among them, ζ and ψ are hyper parameters, and the distance loss r D The distance between the feature sequences before and after the above steps of obtaining the counter feature sequence is expressed as edit distance; the malicious loss r M The malicious nature of the malicious traffic corresponding to the feature sequences before and after the steps of obtaining the counter feature sequence is compared.

[0050] To achieve the above objectives, another object of the present invention is to provide a malicious traffic detection and anti-attack device, comprising:

[0051] A semantic model initialization module, configured to extract a first feature sequence and use the first feature sequence as an input sequence of an initialized traffic semantic model to perform semantic model training;

[0052] A semantic model training module is used to randomly mask part of the input sequence and use the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model;

[0053] an adversarial feature sequence generation module, configured to process the extracted second feature sequence using a deep reinforcement learning model to obtain action data, and to process the second feature sequence based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence;

[0054] A reinforcement learning model training module is used to convert the adversarial feature sequence into adversarial malicious traffic, and send the adversarial malicious traffic from the sending end to the receiving end, obtain feedback information based on the number of data packets successfully received by the receiving end, and combine the feedback information with a preset penalty mechanism to optimize the network of the deep reinforcement learning model to obtain a trained deep reinforcement learning model;

[0055] The model inference module is used to extract features of malicious traffic extracted from the anti-malicious traffic to obtain a third feature sequence; wherein the third feature sequence includes a packet size feature sequence and a packet interval feature sequence of the malicious traffic extracted from the anti-malicious traffic, and the third feature sequence is processed based on the policy network and the trained traffic semantic model to obtain a final anti-malicious feature sequence, so as to convert the final anti-malicious feature sequence into a final anti-malicious traffic.

[0056] The present invention provides a method and apparatus for detecting malicious traffic and countering attacks, involving two phases: benign traffic cloning and adversarial traffic generation. In the benign traffic cloning phase, a traffic semantic model designed specifically for network traffic is trained and deployed to learn the sending patterns of massive amounts of benign traffic on the internet. In the adversarial traffic generation phase, based on feedback from a target detection system and combining deep reinforcement learning with a traffic semantic model, the system embeds the local sending patterns of benign traffic into malicious traffic to generate adversarial traffic. A specific penalty mechanism is also introduced to ensure that the generated traffic retains its original properties.

[0057] The beneficial effects of the present invention are:

[0058] The proposed black-box adversarial attack solution, based on traffic semantic models and deep reinforcement learning, can effectively generate practical adversarial malicious traffic, providing a powerful tool for verifying and improving the robustness of malicious traffic detection systems. Simulating real-world adversarial attack scenarios can also help researchers identify potential weaknesses in existing detection systems, leading to the design of more effective defense systems and technical solutions, ultimately improving network security protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0060] Figure 1 Flowchart of a method for detecting malicious traffic and resisting attacks according to an embodiment of the present invention;

[0061] Figure 2 A logical diagram of a method for detecting and countering malicious traffic attacks according to an embodiment of the present invention;

[0062] Figure 3 Schematic diagram of the principle of a method for detecting malicious traffic and resisting attacks according to an embodiment of the present invention;

[0063] Figure 4 Schematic diagram of the structure of a malicious traffic detection and anti-attack device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0065] The following describes a method and apparatus for detecting malicious traffic and countering attacks according to an embodiment of the present invention with reference to the accompanying drawings.

[0066] The embodiment of the present invention is directed to a black box counterattack against a malicious traffic detection system based on a decision tree and performing detection in flow units. Malicious traffic is malware traffic, and its maliciousness is reflected in the payload.

[0067] The present invention combines traffic semantic models with deep reinforcement learning technology to design a black box adversarial attack technology for malicious traffic detection systems based on machine learning architecture. This allows the present invention to generate adversarial malicious traffic that can escape detection and maintain its original traffic properties without knowing the specific parameters and implementation details of the target detection system. The present invention as a whole can be divided into two stages: the benign traffic cloning stage and the adversarial traffic generation stage. In the benign traffic cloning stage, a traffic semantic model Traffic-BERT suitable for the network field is designed based on the BERT model, and massive benign traffic data on the Internet is collected to learn the sending pattern of benign traffic; in the adversarial traffic generation stage, based on the policy network, Q network and Traffic-BERT, it is determined how to effectively embed benign traffic patterns into malicious traffic, and the generation strategy is optimized by integrating the feedback of the target detection system and a specific penalty mechanism to ensure that the adversarial malicious traffic is difficult to detect and maintains the necessary original traffic properties.

[0068] Figure 1 The figure is a flow chart of a method for detecting malicious traffic and resisting attacks according to an embodiment of the present invention.

[0069] like Figure 1 、 Figure 2 and Figure 3 As shown, the method includes but is not limited to the following steps:

[0070] S1: extract a first feature sequence and use the first feature sequence as an input sequence of an initialized traffic semantic model.

[0071] In this stage, benign traffic cloning is performed. In this stage, benign Internet traffic is collected as training data and data processing is performed. The data processing method includes extracting packet feature sequences and packet interval feature sequences, and filling and blocking the extracted feature sequences.

[0072] The first feature sequence extracted in this embodiment is extracted based on collected benign internet traffic. Benign internet traffic can be collected by downloading data from any open-source internet traffic dataset or capturing traffic data from the internet, followed by data preprocessing to remove any obvious abnormal data.

[0073] The method for removing obviously abnormal data is to identify and exclude data with obvious malicious traffic characteristics. For example, this includes filtering out traffic from the same IP address scanning a large number of random addresses, traffic with abnormally high sending rates that may indicate a DoS attack, and other traffic with specific malicious characteristics.

[0074] The first feature sequence, namely the packet size feature sequence and packet interval feature sequence of benign internet traffic data, is extracted by dividing all data packets into flows. A flow can be defined as a five-tuple consisting of <source IP address, destination IP address, source port, destination port, protocol> or other reasonable definitions. The packet size feature sequence is extracted per flow, with each element in the sequence representing the size of the corresponding data packet. The packet interval feature sequence is extracted per flow, with each element in the sequence representing the time interval between the transmission of the corresponding data packet and the previous data packet, with the first element set to 0.

[0075] Furthermore, the method of padding and blocking the extracted packet size feature sequence and packet interval feature sequence is to set the fixed length of the feature sequence to n=512, and perform padding processing on the feature sequence with a length less than n, that is, add a number of zero values ​​at the end of the sequence until the fixed length n is reached; and perform blocking processing on the feature sequence with a length greater than n, that is, starting from each element of the sequence, sequentially intercept subsequences of length n as new sequences until it is impossible to continue to intercept new sequences that meet the fixed length requirements from the original sequence.

[0076] Furthermore, this step initializes the traffic semantic model Traffic-BERT, including initializing the model structure and parameters.

[0077] The Traffic-BERT model in this embodiment of the present invention consists of several stacked encoder layers. Each encoder includes a self-attention layer, a bidirectional-cross attention layer, and a feedforward neural network layer. The different layers are tightly connected through residual connections and layer normalization. Traffic-BERT uses a sequence of packet size features and a sequence of packet interval features as model input, and processes the data through a self-attention layer, a bidirectional-cross attention layer, and a feedforward neural network layer.

[0078] The first feature sequence is input into the self-attention layer. For a given single input sequence, the self-attention layer calculates the attention weight of each element in the sequence with respect to all other elements, and generates a weighted output sequence based on these weights. The calculation process can be described as follows: each element of the input sequence is converted into three vectors - query vector Q, key vector K and value vector V. These three vectors are obtained by linear mapping with the input elements; based on these three vectors, the output of the self-attention layer is:

[0079]

[0080] Among them, d k is the dimension of the key vector.

[0081] Among them, the two equal-length input sequences in the first feature sequence are input into the symmetrical cross-attention layer; the bidirectional-cross-attention layer is composed of a symmetrical cross-attention layer, which receives two equal-length sequences as input. Its calculation process can be expressed as: the first input sequence is converted into a query vector Q1, a key vector K1 and a value vector V1, and the second input sequence is converted into a query vector Q2, a key vector K2 and a value vector V2. The output of the bidirectional-cross-attention layer corresponding to the first input sequence is Attn(Q1, K2, V2), and the output of the bidirectional-cross-attention layer corresponding to the second input sequence is Attn(Q2, K1, V1).

[0082] Furthermore, the first output sequence Attn(Q1, K2, V2) and the second output sequence Attn(Q2, K1, V1) are input into the feedforward neural network layer to obtain the final output sequence.

[0083] It can be understood that residual connection means that the input of the network skips a certain layer and is directly added to the subsequent layer; layer normalization is used to perform feature normalization within each layer of the network, which can be expressed as:

[0084]

[0085] Among them, μ and σ are the mean and standard deviation of all neurons in this layer, and γ and β are learnable parameters.

[0086] Furthermore, Traffic-BERT is initialized using normal distribution random initialization, thereby completing the initialization of the Traffic-BERT model.

[0087] S2, randomly masking part of the sequence in the input sequence, and using the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model.

[0088] It is understandable that this step trains Traffic-BERT. After Traffic-BERT training is completed, the parameters of Traffic-BERT are fixed to complete benign traffic cloning.

[0089] It's understandable that the Traffic-BERT model is trained as a masked language model. This involves randomly masking portions of the input sequence and then having the model predict the original values ​​of these masked portions. Specifically, a preset number (p%) of elements in the packet size and packet interval feature sequences are randomly masked, replacing them with a special [MASK] tag. Traffic-BERT then predicts the values ​​of the masked elements.

[0090] Furthermore, based on the above prediction results, Traffic-BERT is optimized using forward propagation and backpropagation algorithms. The optimizer used in training can be the Adam optimizer, and the loss function can be the cross-entropy loss function.

[0091] In one embodiment of the present invention, Traffic-BERT training is determined to be complete when the number of training rounds exceeds the target number of rounds Ep, or when the model performance no longer improves within the number of rounds El.

[0092] S3, using the deep reinforcement learning model to process the extracted second feature sequence to obtain action data, and based on the action data and the trained traffic semantic model, processing the second feature sequence to obtain an adversarial feature sequence.

[0093] As you can see, this stage is about generating adversarial traffic. This stage mainly establishes the deep reinforcement learning framework, initializes the policy network and Q network, and begins training the deep reinforcement learning model.

[0094] It can be understood that establishing a deep reinforcement learning framework means establishing a deep reinforcement learning framework with the Soft Actor-Critic (SAC) algorithm as the main body, and simultaneously learning a policy network and a Q network to maximize the long-term reward and entropy of the policy.

[0095] The policy network and Q network of the deep reinforcement learning model in this embodiment of the present invention are based on Gated Recurrent Unit (GRU) networks, which take as input a concatenation of a packet size feature sequence and a packet interval feature sequence. The policy network outputs a discrete value indicating the action at time step t; the Q network outputs a vector that evaluates the state-action value at time step t.

[0096] Among them, the policy network and Q network are initialized by normal distribution random initialization.

[0097] If the deep reinforcement learning training environment is not initialized, initialize it; otherwise, reset the environment to its initial state.

[0098] The deep reinforcement learning training environment includes the malicious traffic sender, the target malicious traffic detection system, and the malicious traffic receiver. The target detection system is a deployed detection system, and its internal implementation details cannot be observed. The sender and receiver are located upstream and downstream of the target detection system, respectively, and malicious traffic can be detected based on the traffic arriving at the receiver. Initializing the environment involves setting the sender and receiver to normal working conditions and extracting a malicious flow from the pre-prepared malicious traffic data.

[0099] Furthermore, the present invention performs feature processing on malicious traffic, extracting a second feature sequence: the packet size feature sequence and the packet interval feature sequence of the malicious traffic data, and performing length processing. The policy network receives the feature sequence and takes an action. The action is combined with Traffic-BERT to process the feature sequence, generating an adversarial feature sequence.

[0100] Exemplarily, the manner of extracting the second feature sequence, the packet size feature sequence and the packet interval feature sequence is consistent with that described above and will not be described in detail here.

[0101] For example, the second feature sequence is length-processed by extracting a subsequence of the first n elements of the sequence. If the sequence length is less than n, the original sequence is padded according to the same steps as above, which will not be described in detail here.

[0102] Exemplarily, a subsequence of the second feature sequence is intercepted and connected end to end to form a feature sequence of a second preset length to be input into the policy network of the deep reinforcement learning model; specifically, the policy network receives the feature sequence by splicing the packet size feature sequence and the packet interval feature sequence with a length of n end to end to form a feature sequence with a length of 2n.

[0103] Exemplarily, the output action of the policy network is a positive integer with a value range of [0, 2n].

[0104] This embodiment of the present invention first adjusts the malicious signature sequence based on the action value output by the policy network: Assuming the action is a, if a is an odd number, the value at position (a-1) / 2 in the packet interval signature sequence is set to [MASK], while the packet size sequence remains unchanged. If a is an even number, a [MASK] is added between the packet size sequence and the packet interval sequence at positions a / 2 and a / 2+1 to form a new signature sequence. These two new signature sequences are then fed into Traffic-BERT, and the predicted value at the [MASK] position is replaced by Traffic-BERT, resulting in an adversarial signature sequence.

[0105] S4, converting the adversarial feature sequence into adversarial malicious traffic, and sending the adversarial malicious traffic from the sending end to the receiving end, obtaining feedback information based on the number of data packets successfully received by the receiving end, combining the feedback information with a preset penalty mechanism, and optimizing the network of the deep reinforcement learning model to obtain a trained deep reinforcement learning model.

[0106] The embodiment of the present invention converts the confrontation feature sequence into confrontation malicious traffic, sends it from the sending end to the receiving end, obtains feedback based on the number of data packets successfully received by the receiving end, and combines the feedback with a specific penalty mechanism to optimize the policy network and Q network.

[0107] Specifically, the method of converting the countermeasure feature sequence into the countermeasure malicious traffic is to directly restore the original data packet for the packet size sequence feature that has not been adjusted after step S4; otherwise, the feature is a newly added feature, and a new data packet should be added according to the packet size countermeasure sequence feature, and the correct header information should be filled in according to other data packets, and a random payload should be filled in to ensure that the size of the newly added data packet is equal to the newly added feature. According to the packet interval countermeasure sequence feature, the timestamp of all subsequent data packets is recalculated based on the timestamp of the first data packet. The lifetime field of the header of the newly added data packet is adjusted so that it can pass the malicious traffic detection system, but will not be received by the receiving end, thereby ensuring that the payload information of the original malicious flow remains unchanged.

[0108] The method for obtaining feedback based on the number of data packets successfully received by the receiving end is to calculate the number of valid malicious data packets successfully received by the receiving end, and divide the number by n as the feedback value r E .

[0109] Among them, the specific penalty mechanism includes distance loss r D and malicious loss M The way to combine feedback with a specific punishment mechanism is to add the three together, namely:

[0110] r=r E +ζr D +ψr M

[0111] Among them, ζ = 0.1 and ψ = 0.1 are hyperparameters.

[0112] Among them, the distance loss r D It is used to measure the distance between the feature sequences before and after the above steps of obtaining the adversarial feature sequence, expressed as edit distance. Since the original feature sequence and the adversarial feature sequence at any time step have only one difference, so:

[0113] r D =-1

[0114] Among them, malicious loss rM It is used to measure the degree to which the feature sequence before and after the steps of obtaining the counter feature sequence destroys the original malicious traffic. For example, for DoS traffic, r M It can be defined as the difference between the sending rates of the previous and next feature sequences.

[0115] The method for determining whether the trajectory of combating malicious traffic has ended is to determine whether the feedback value r E Is it greater than the threshold r ε = 0.99. If yes, the trajectory ends, otherwise it does not end.

[0116] If it has not yet concluded, the modified adversarial malicious flow is used as the new malicious flow, and the process returns to step S4, where the malicious flow is processed to obtain an adversarial feature sequence. Otherwise, the process determines whether the reinforcement learning model training has concluded. The method for determining whether the reinforcement learning model training has concluded can be to determine whether the number of training trajectories exceeds the target number of trajectories Epi = 2000. If it has concluded, proceed to the next step. Otherwise, return to step S4, and if the environment has not been initialized, initialize it; otherwise, reset the environment to its initial state.

[0117] Furthermore, feature extraction is performed on the malicious traffic extracted from the anti-malicious traffic to obtain a third feature sequence; wherein, the third feature sequence includes a packet size feature sequence and a packet interval feature sequence of the malicious traffic extracted from the anti-malicious traffic; the third feature sequence is processed based on the policy network and the trained traffic semantic model to obtain a final anti-malicious feature sequence; and the final anti-malicious feature sequence is converted into a final anti-malicious traffic.

[0118] Specifically, after training the aforementioned model, adversarial malicious traffic inference begins. Malicious flows are extracted from the malicious traffic data, and the third feature sequence, the packet size feature sequence, and the packet interval feature sequence are extracted and length processed. The feature sequence is repeatedly processed using the policy network and Traffic-BERT until the generation process is complete, ultimately generating an adversarial feature sequence. This adversarial feature sequence is converted into adversarial malicious traffic, concluding the adversarial traffic generation phase.

[0119] The method of extracting the packet size feature sequence and the packet interval feature sequence is consistent with the above steps and will not be described in detail here.

[0120] The length processing method is the same as the above steps and will not be described in detail here.

[0121] The method of combining the policy network and Traffic-BERT to process the feature sequence is consistent with the above steps and will not be repeated here.

[0122] The method for judging the end of the generation process is to determine the end of the generation process according to the action a at time step t. t , corresponding to the Q network output Is it greater than the target threshold Q T , or the time step t is greater than a target threshold τ, if so, the generation process ends.

[0123] The malicious traffic detection and counter-attack method of the embodiment of the present invention first trains and deploys a traffic semantic model designed specifically for network traffic to learn the sending patterns of massive benign traffic on the Internet. Secondly, a specific penalty mechanism is introduced to ensure the original traffic properties of the generated traffic. Deep reinforcement learning technology is applied to process the feedback of the target detection system on malicious traffic, and combined with the traffic semantic model, counter-malicious traffic that can evade detection is generated.

[0124] Furthermore, if Figure 4 As shown, in order to implement the above embodiment, the present invention proposes a malicious traffic detection and anti-attack device 10, which includes:

[0125] The semantic model initialization module 100 is used to randomly mask a portion of the input sequence and use the initialized traffic semantic model to predict the original value of the masked portion of the sequence to train the traffic semantic model to obtain a trained traffic semantic model;

[0126] The semantic model training module 200 is used to randomly mask a portion of the input sequence and use the initialized traffic semantic model to predict the original value of the masked portion of the sequence to train the traffic semantic model to obtain a trained traffic semantic model;

[0127] An adversarial feature sequence generation module 300 is configured to process the extracted second feature sequence using a deep reinforcement learning model to obtain action data, and to process the second feature sequence based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence;

[0128] A reinforcement learning model training module 400 is configured to convert the adversarial feature sequence into adversarial malicious traffic, send the adversarial malicious traffic from the transmitting end to the receiving end, obtain feedback information based on the number of data packets successfully received by the receiving end, and combine the feedback information with a preset penalty mechanism to optimize the network of the deep reinforcement learning model to obtain a trained deep reinforcement learning model;

[0129] The model inference module 500 is used to extract features from the malicious traffic extracted from the anti-malicious traffic to obtain a third feature sequence; wherein the third feature sequence includes a packet size feature sequence and a packet interval feature sequence of the malicious traffic extracted from the anti-malicious traffic; the third feature sequence is processed based on the policy network and the trained traffic semantic model to obtain a final anti-malicious feature sequence; and the final anti-malicious feature sequence is converted into a final anti-malicious traffic.

[0130] The malicious traffic detection and anti-attack device of the embodiment of the present invention first trains and deploys a traffic semantic model designed specifically for network traffic to learn the sending pattern of massive benign traffic on the Internet, and then introduces a specific penalty mechanism to ensure the original traffic properties of the generated traffic. Deep reinforcement learning technology is applied to process the feedback of the target detection system on malicious traffic, and combined with the traffic semantic model, anti-malicious traffic that can evade detection is generated.

[0131] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0132] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0133] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for detecting malicious traffic and resisting attacks, characterized in that: include: Extracting a first feature sequence, and using the first feature sequence as an input sequence of an initialized traffic semantic model; Randomly masking part of the sequence in the input sequence, and using the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model; Using a deep reinforcement learning model to process the extracted second feature sequence to obtain action data, and processing the second feature sequence based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence; Converting the adversarial feature sequence into adversarial malicious traffic, and sending the adversarial malicious traffic from the sending end to the receiving end, obtaining feedback information based on the number of data packets successfully received by the receiving end, and combining the feedback information with a preset penalty mechanism to optimize the network of the deep reinforcement learning model to obtain a trained deep reinforcement learning model; in, The first feature sequence includes a packet size feature sequence and a packet interval feature sequence of benign Internet traffic data; the second feature sequence includes a packet size feature sequence and a packet interval feature sequence of preset malicious traffic data.

2. The method according to claim 1, characterized in that The traffic semantic model is composed of several layers of encoder stacks; each encoder includes a self-attention layer, a bidirectional-cross attention layer and a feedforward neural network layer; the bidirectional-cross attention layer is composed of symmetrical cross attention layers, and different layers are connected by residual connections and layer normalization; the residual connection is used to skip a certain layer of the input of the traffic semantic model network and directly add it to the subsequent layer; the layer normalization is used to perform feature normalization processing within each layer of the network; the deep reinforcement learning model is used for training an environment including the sender of malicious traffic, the target malicious traffic detection system, and the receiver of malicious traffic.

3. The method according to claim 2, characterized in that The deep reinforcement learning model includes a policy network and a Q network; After obtaining the trained deep reinforcement learning model, the method further includes: Extracting features from the malicious traffic extracted from the anti-malicious traffic to obtain a third feature sequence; wherein the third feature sequence includes a packet size feature sequence and a packet interval feature sequence of the malicious traffic extracted from the anti-malicious traffic; Processing the third feature sequence based on the policy network and the trained traffic semantic model to obtain a final adversarial feature sequence; The final adversarial feature sequence is converted into the final adversarial malicious traffic.

4. The method according to claim 1, wherein Extracting the first feature sequence includes: Downloading traffic data from an open source Internet traffic dataset, or capturing traffic data from the Internet, and performing data preprocessing on the traffic data to obtain preprocessed data; The preprocessed data is divided into stream units, and a packet size feature sequence and a packet interval feature sequence of the preprocessed data are extracted in units of streams to obtain a first feature sequence; wherein each element in the packet size feature sequence of the preprocessed data is the size of a corresponding data packet, and each element in the packet interval feature sequence of the preprocessed data is the time interval between the transmission of the corresponding data packet and the previous data packet; The extracted first feature sequence is padded and divided into blocks.

5. The method according to claim 4, characterized in that The extracted first feature sequence is filled and divided into blocks, including: Determining a first preset length of the first feature sequence, and performing padding processing on feature sequences whose length is less than the first preset length, that is, adding a number of zero values ​​at the end of the sequence until the preset length is reached; For feature sequences whose length is greater than the first preset length, block processing is performed, that is, starting from each element of the sequence, subsequences with the first preset length are sequentially intercepted as new sequences until no new sequences that meet the first preset length requirements can be further intercepted from the original sequence.

6. The method according to claim 1, characterized in that Taking the first feature sequence as an input sequence, comprising: Inputting the first feature sequence into a self-attention layer to calculate the attention weight of each element in a single input sequence with respect to all other elements, and generating a weighted output sequence based on the attention weights; Inputting two equal-length input sequences in the first feature sequence into a symmetrical cross-attention layer based on the weighted output sequence to convert the first input sequence into a first query vector, a first key vector, and a first value vector; and converting the second input sequence into a second query vector, a second key vector, and a second value vector; Obtain a first output sequence based on the first query vector, the second key vector, and the second value vector, and obtain a second output sequence based on the second query vector, the first key vector, and the first value vector; The first output sequence and the second output sequence are input into the feed-forward neural network layer to obtain the final output sequence.

7. The method according to claim 1, characterized in that Randomly masking part of the input sequence, and using the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model, including: Randomly masking a preset number of elements in the first feature sequence and replacing the preset number of elements with mask marks to predict the masked element values ​​through the traffic semantic model; According to the prediction results, the traffic semantic model is optimized using forward propagation and back propagation algorithms to obtain a trained traffic semantic model.

8. The method according to claim 2, characterized in that The extracted second feature sequence is processed using a deep reinforcement learning model to obtain action data, and the second feature sequence is processed based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence, including: Extracting a packet size feature sequence and a packet interval feature sequence of preset malicious traffic data to obtain a second feature sequence; Extracting subsequences of the second feature sequence and concatenating them end to end to form a feature sequence of a second preset length for input into the policy network of the deep reinforcement learning model; The output action value of the strategy network is a positive integer, and the preset action value is ,like is an odd number, the packet interval characteristic sequence of the second preset length characteristic sequence Mask; if If it is an even number, the packet size feature sequence and packet interval feature sequence of the second preset length feature sequence are and Add a mask between positions to form a new feature sequence; The two new feature sequences are input into the trained traffic semantic model, and the predicted value of the trained traffic semantic model for the mask position replaces the mask to obtain the adversarial feature sequence.

9. The method according to claim 8, characterized in that Converting the countermeasure feature sequence into countermeasure malicious traffic, sending the countermeasure malicious traffic from the sending end to the receiving end, and obtaining feedback information based on the number of data packets successfully received by the receiving end, including: For packet size adversarial sequence features that are not adjusted in the characteristic sequence of the second preset length, directly restore the original data packet; otherwise, add a new data packet based on the packet size adversarial sequence features, fill in the correct packet header information for the other data packets except the newly added data packet, and fill in the random payload; Based on the packet interval countermeasure sequence feature, the timestamp of all subsequent data packets is recalculated based on the timestamp of the first data packet; Adjust the Time to Live field in the header of newly added data packets so that they are discarded after passing through the malicious traffic detection system and are not received by the receiving end; The number of valid malicious data packets successfully received by the receiving end is calculated, and a feedback value is obtained based on the number of valid malicious data packets.

10. The method according to claim 9, characterized in that The method further comprises: The feedback value The combination is the sum of the three, namely: in, and is a hyperparameter, the distance loss The distance between the feature sequences before and after the above steps of obtaining the counter feature sequence is expressed as edit distance; the malicious loss is the comparison of the malicious nature of the malicious traffic corresponding to the feature sequence before and after the above steps of obtaining the counter feature sequence, r is the feedback value r E , distance loss r D and malicious loss M Add up the total losses.

11. A malicious traffic detection and anti-attack device, characterized in that: include: A semantic model initialization module, configured to extract a first feature sequence and use the first feature sequence as an input sequence of an initialized traffic semantic model to perform semantic model training; A semantic model training module is used to randomly mask part of the input sequence and use the initialized traffic semantic model to predict the original value of the masked part of the sequence to train the traffic semantic model to obtain a trained traffic semantic model; an adversarial feature sequence generation module, configured to process the extracted second feature sequence using a deep reinforcement learning model to obtain action data, and to process the second feature sequence based on the action data and the trained traffic semantic model to obtain an adversarial feature sequence; A reinforcement learning model training module is used to convert the adversarial feature sequence into adversarial malicious traffic, and send the adversarial malicious traffic from the sending end to the receiving end, obtain feedback information based on the number of data packets successfully received by the receiving end, and combine the feedback information with a preset penalty mechanism to optimize the network of the deep reinforcement learning model to obtain a trained deep reinforcement learning model; A model inference module is configured to extract features from the malicious traffic extracted from the anti-malicious traffic to obtain a third feature sequence; wherein the third feature sequence includes a packet size feature sequence and a packet interval feature sequence of the malicious traffic extracted from the anti-malicious traffic, and the third feature sequence is processed based on the policy network and the trained traffic semantic model to obtain a final anti-malicious feature sequence, so as to convert the final anti-malicious feature sequence into a final anti-malicious traffic; in, The first feature sequence includes a packet size feature sequence and a packet interval feature sequence of benign Internet traffic data; the second feature sequence includes a packet size feature sequence and a packet interval feature sequence of preset malicious traffic data.

Images