A network traffic data denoising method, device, equipment and storage medium

Abstract

The application discloses a network flow data denoising method, device and equipment and a storage medium, comprising: dividing an original network flow capture tensor into a first flow sample not subjected to a boundary attack and a second flow sample subjected to a boundary attack; training a variational autoencoder using the first flow sample, and encoding the second flow sample using the variational autoencoder; decoding an abnormal flow feature vector using a latent space prior probability model, and inputting a reconstructed network flow tensor and the original network flow capture tensor into a tensor decomposition model; and jointly training the variational autoencoder and the tensor decomposition model according to a reconstruction error between the low-rank reconstructed network flow tensor and the original network flow capture tensor until the reconstruction error satisfies a convergence condition, and outputting a denoised network flow tensor. The technical scheme of the embodiment of the application can effectively denoise network flow data and improve the overall level of network security situation awareness and governance.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for reducing noise in network traffic data. Background Art

[0002] At present, automated monitoring of network traffic data and high-dimensional feature analysis have become key links in the network security protection system. In order to improve network security, it is very necessary to perform noise reduction before analyzing network traffic data.

[0003] In the existing technology, the methods for denoising network traffic data mainly involve the following: First, methods based on manual feature filtering or threshold segmentation, which screen and clean specific traffic features through preset rules, but are difficult to adapt to the complex correlations and dynamic changes of network traffic in high-dimensional multimodal environments, and have limited real-time and generalization capabilities; Second, traditional deep generative models are used to reconstruct and filter traffic data, which has certain effects in processing conventional noise, but when faced with boundary attack traffic with complex high-order correlation structures and distribution drift characteristics, the reconstruction results often have distribution distortion and reduced discrimination ability.

[0004] Therefore, there is an urgent need for a new network traffic denoising method that can take into account multi-dimensional structural correlation, distribution consistency and anti-disturbance identification capabilities, so as to improve the level of network security situation awareness and automated governance. Summary of the Invention

[0005] The present invention provides a network traffic data noise reduction method, device, equipment and storage medium, which can effectively reduce the noise of network traffic data and improve the overall level of network security situation awareness and management.

[0006] According to one aspect of the present invention, a method for reducing noise in network traffic data is provided, the method comprising:

[0007] Obtaining an original network traffic capture tensor corresponding to the network interface, and dividing the original network traffic capture tensor into a first traffic sample that has not been attacked by the boundary, and a second traffic sample that has been attacked by the boundary;

[0008] The first traffic sample is used to train the variational autoencoder to obtain a latent space prior probability model, and the second traffic sample is encoded using the variational autoencoder to obtain an abnormal traffic feature vector;

[0009] Using a latent space prior probability model to decode the abnormal traffic feature vector to obtain a reconstructed network traffic tensor, inputting the reconstructed network traffic tensor and the original network traffic capture tensor into a preset tensor decomposition model, and outputting a low-rank reconstructed network traffic tensor through the tensor decomposition model;

[0010] According to the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic captured tensor, the variational autoencoder and the tensor decomposition model are jointly trained until the reconstruction error meets the preset convergence condition and the denoised network traffic tensor is output.

[0011] According to another aspect of the present invention, a network traffic data noise reduction device is provided, the device comprising:

[0012] a traffic collection module, configured to obtain a raw network traffic capture tensor corresponding to a network interface, and divide the raw network traffic capture tensor into a first traffic sample that has not been attacked by a boundary, and a second traffic sample that has been attacked by a boundary;

[0013] A sample processing module is used to train a variational autoencoder using the first traffic sample to obtain a latent space prior probability model, and to encode the second traffic sample using the variational autoencoder to obtain an abnormal traffic feature vector;

[0014] A vector decoding module is used to decode the abnormal traffic feature vector using a latent space prior probability model to obtain a reconstructed network traffic tensor, input the reconstructed network traffic tensor and the original network traffic capture tensor into a preset tensor decomposition model, and output a low-rank reconstructed network traffic tensor through the tensor decomposition model;

[0015] The joint training module is used to jointly train the variational autoencoder and the tensor decomposition model according to the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor until the reconstruction error meets the preset convergence condition and outputs the denoised network traffic tensor.

[0016] According to another aspect of the present invention, an electronic device is provided, comprising:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the network traffic data denoising method described in any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the network traffic data denoising method described in any embodiment of the present invention when executed.

[0021] According to another aspect of the present invention, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the network traffic data denoising method according to any embodiment of the present invention is implemented.

[0022] The technical solution provided by the embodiment of the present invention obtains the original network traffic capture tensor corresponding to the network interface, divides the original network traffic capture tensor into a first traffic sample that has not been attacked by a boundary, and a second traffic sample that has been attacked by a boundary, uses the first traffic sample to train the variational autoencoder to obtain a latent space prior probability model, and uses the variational autoencoder to encode the second traffic sample to obtain an abnormal traffic feature vector, uses the latent space prior probability model to decode the abnormal traffic feature vector to obtain a reconstructed network traffic tensor, inputs the reconstructed network traffic tensor and the original network traffic capture tensor into a preset tensor decomposition model, outputs a low-rank reconstructed network traffic tensor through the tensor decomposition model, and jointly trains the variational autoencoder and the tensor decomposition model according to the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor until the reconstruction error meets the preset convergence condition, and outputs the denoised network traffic tensor. The technical means can effectively denoise network traffic data and improve the overall level of network security situation awareness and governance.

[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 This is a flow chart of a method for reducing noise in network traffic data provided according to an embodiment of the present invention;

[0026] Figure 2 is a flow chart of another method for reducing noise in network traffic data provided according to an embodiment of the present invention;

[0027] Figure 3 2. It is a structural diagram of a network traffic data noise reduction device provided according to an embodiment of the present invention;

[0028] Figure 4 It is a structural diagram of an electronic device for implementing the network traffic data noise reduction method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Figure 1 This is a flow chart of a network traffic data noise reduction method provided by an embodiment of the present invention. This embodiment is applicable to the case where noise reduction is performed before analyzing network traffic data. The method can be performed by a network traffic data noise reduction device, which can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0032] Step 110: Obtain an original network traffic capture tensor corresponding to the network interface, and divide the original network traffic capture tensor into a first traffic sample that has not been attacked by a boundary, and a second traffic sample that has been attacked by a boundary.

[0033] In this step, specifically, the network interface can be monitored in real time to obtain the original network traffic capture tensor, and then the boundary attack suffered by the original network traffic capture tensor is identified. According to the identification results, it is divided into normal network traffic samples that have not been attacked by the boundary (that is, the first traffic sample), and abnormal network traffic samples that may have been attacked by the boundary (that is, the second traffic sample).

[0034] Step 120: Use the first traffic sample to train the variational autoencoder to obtain a latent space prior probability model, and use the variational autoencoder to encode the second traffic sample to obtain an abnormal traffic feature vector.

[0035] In this step, the first traffic sample can be input into the variational autoencoder, and the features of the first traffic sample can be extracted by the variational autoencoder. Then, the parameters of the variational autoencoder can be optimized according to the feature extraction results to train a latent space prior probability model. Then, the variational autoencoder is used to perform an encoding operation on the second traffic sample to generate a latent space feature representation of the abnormal traffic sample (i.e., an abnormal traffic feature vector) that is consistent with the latent space prior probability model.

[0036] Step 130: Use the latent space prior probability model to decode the abnormal traffic feature vector to obtain a reconstructed network traffic tensor, input the reconstructed network traffic tensor and the original network traffic capture tensor into a preset tensor decomposition model, and output a low-rank reconstructed network traffic tensor through the tensor decomposition model.

[0037] In this step, the latent space prior probability model can be used to decode the abnormal traffic feature vector to obtain a reconstructed network traffic tensor, and then the reconstructed network traffic tensor and the original network traffic capture tensor are input into the distribution reconstruction tensor decomposition model, and low-rank tensor decomposition is performed under the condition of multi-dimensional structure preservation to obtain a low-rank reconstructed network traffic tensor.

[0038] The advantage of this setting is that by using the latent space prior probability model to decode the abnormal traffic feature vector, the initial structure of the abnormal traffic can be restored, assisting in removing the boundary attack disturbance noise. Since the second traffic sample has been encoded as a feature representation vector located in the latent space, it is generated based on the latent space prior probability model obtained by training the first traffic sample and has good "normal traffic statistical distribution" characteristics. By inputting the abnormal traffic feature vector into the decoder for decoding, it is equivalent to projecting the disturbed abnormal traffic back to the distribution domain of normal traffic, thereby eliminating the abnormal high-frequency features or noise components introduced by the attack disturbance.

[0039] Secondly, the above decoding operation can also provide a compressible and structurally aligned input basis for the subsequent low-rank tensor decomposition. Although the "reconstructed network traffic tensor" obtained by decoding already has certain anti-interference properties, it still contains some residual adversarial disturbance signals. In order to further extract the essential structure, compress redundant information and restore the true statistical distribution, the low-rank tensor decomposition mechanism was subsequently introduced. The decomposition model requires that the input tensor must be structurally aligned and close to the normal traffic shape in distribution, otherwise the decomposition result will be distorted or convergence failure.

[0040] Step 140: Based on the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor, the variational autoencoder and the tensor decomposition model are jointly trained until the reconstruction error meets the preset convergence condition, and the denoised network traffic tensor is output.

[0041] In this step, specifically, the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor can be determined, the noise energy threshold can be determined based on the reconstruction error, and then the latent space prior probability model parameters and the tensor decomposition model parameters can be reversely updated based on the noise energy threshold to perform joint adaptive optimization of the variational autoencoder and the tensor decomposition model, that is, return to execute the operations of steps 120 to 130 until the reconstruction error meets the preset convergence conditions, and the final low-rank reconstructed network traffic tensor is output as the denoised network traffic tensor.

[0042] The advantage of this setting is that by implementing closed-loop optimization of the parameters of the variational autoencoder and tensor decomposition model in each round of iteration, the noise reduction effect, adaptability and scenario generalization ability of the variational autoencoder can be improved. It can also ensure the real-time processing performance of the variational autoencoder in complex network environments and high-traffic and high-concurrency backgrounds, thereby improving the level of network security situation awareness and automated governance.

[0043] The technical solution provided by the embodiment of the present invention obtains the original network traffic capture tensor corresponding to the network interface, divides the original network traffic capture tensor into a first traffic sample that has not been attacked by a boundary, and a second traffic sample that has been attacked by a boundary, uses the first traffic sample to train the variational autoencoder to obtain a latent space prior probability model, and uses the variational autoencoder to encode the second traffic sample to obtain an abnormal traffic feature vector, uses the latent space prior probability model to decode the abnormal traffic feature vector to obtain a reconstructed network traffic tensor, inputs the reconstructed network traffic tensor and the original network traffic capture tensor into a preset tensor decomposition model, outputs a low-rank reconstructed network traffic tensor through the tensor decomposition model, and jointly trains the variational autoencoder and the tensor decomposition model according to the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor until the reconstruction error meets the preset convergence condition, and outputs the denoised network traffic tensor. The technical means can effectively denoise network traffic data and improve the overall level of network security situation awareness and governance.

[0044] Figure 2 A flowchart of another network traffic data noise reduction method provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the method includes:

[0045] Step 210: Obtain the original network traffic capture tensor corresponding to the network interface, determine the boundary attack probability for each packet of the original network traffic capture tensor through a preset boundary attack discrimination model, and output the boundary attack discrimination result corresponding to the original network traffic capture tensor based on the boundary attack probability.

[0046] In one implementation of this embodiment, obtaining the original network traffic capture tensor corresponding to the network interface includes:

[0047] Step 211: Collect traffic packet data corresponding to the network interface in real time;

[0048] Step 212: Format the timestamp dimension, protocol stack dimension, topology dimension feature, and content dimension feature contained in the traffic packet data to obtain an original network traffic capture tensor;

[0049] In this step, the timestamp dimension, protocol stack dimension, topology dimension and content dimension features contained in all traffic packet data can be uniformly formatted and mapped into a four-dimensional original network traffic capture tensor, which is defined in a four-dimensional space composed of the number of time windows, the number of protocol categories, the number of topology path combinations and the number of content features. Each element in the four-dimensional original network traffic capture tensor is used to represent the traffic value obtained by statistics at a specific time slice, protocol type, topology combination and content feature position.

[0050] Among them, the timestamp dimension is used to characterize the capture moment of the traffic packet data; the protocol stack dimension is used to characterize the transmission protocol type; the topology dimension feature is used to characterize the association relationship between the source node and the target node; and the content dimension feature is used to characterize the load content feature information.

[0051] Step 213: Construct the position value of each tensor unit in the original network traffic capture tensor based on the discretized time window, protocol category set, topology path set, and content feature set.

[0052] In this step, the position value is used to describe the specific numerical indicators counted under a specific time window, protocol type, topology path combination, and content characteristics, and is used to express the traffic intensity of the multi-dimensional combination in the original network traffic.

[0053] In one implementation of this embodiment, a preset boundary attack discrimination model is used to determine the boundary attack probability for each packet of the original network traffic capture tensor, and a boundary attack discrimination result corresponding to the original network traffic capture tensor is output based on the boundary attack probability, including:

[0054] Step 214: extract the position value of the tensor unit at the corresponding dimension position for each network traffic packet sample in the original network traffic capture tensor;

[0055] Step 215: Construct a boundary attack discrimination model. The model is a multi-layer feedforward neural network structure, including an input layer, a hidden layer, and an output layer. The input layer is used to receive the position value of each network traffic packet sample tensor unit. The hidden layer consists of two parallel channels, one of which is used to extract the traffic feature distribution of normal network traffic samples, and the other is used to capture the high-frequency disturbance trajectory characteristics of boundary disturbance samples. The output layer uses an activation function to calculate the boundary attack probability, which is used to represent the probability value of the sample being judged as a boundary attack disturbance sample.

[0056] Step 216: Construct a training set to train the boundary attack discrimination model. The training set includes a set of manually labeled normal network traffic samples and a set of boundary attack perturbation samples generated by the boundary attack algorithm. During the training process, a loss function minimization strategy is adopted to optimize the parameters of the boundary attack discrimination model. After the training is completed, a boundary attack discrimination model with fixed parameters is obtained.

[0057] Step 217: In the boundary attack discrimination model, the boundary attack probability of each network traffic packet sample is compared with the set discrimination threshold. If the boundary attack probability is greater than or equal to the discrimination threshold, the corresponding network traffic packet sample is determined to be a sample that may be affected by the boundary attack, and all network traffic packet samples that meet the conditions are combined into an abnormal traffic set.

[0058] Step 218: All network traffic packet samples whose boundary attack probability is less than the discrimination threshold are determined to be normal network traffic samples, and the normal network traffic samples constitute a normal network traffic sample set.

[0059] The advantage of this setting is that by building a boundary attack discrimination model, the boundary attack corresponding to the original network traffic capture tensor can be quickly identified to improve the efficiency of subsequent network data noise reduction.

[0060] Step 220: According to the boundary attack identification result, the original network traffic capture tensor is divided into a first traffic sample that has not been subjected to the boundary attack and a second traffic sample that has been subjected to the boundary attack.

[0061] Step 230: Use a variational autoencoder to extract the mean vector and standard deviation vector corresponding to the first traffic sample; and generate a feature representation vector corresponding to each first traffic sample in the latent space based on the mean vector and standard deviation vector.

[0062] In this embodiment, the variational autoencoder consists of a conditional attention encoder network and a distribution-guided decoder network.

[0063] In this step, specifically, each first traffic sample can be input into the conditional attention encoder network, and its feature expression is extracted based on the tensor position values ​​on the time dimension, protocol dimension, topology dimension and content dimension contained in the input sample, and the sensitivity of the sample to boundary disturbances in different dimensions is considered. Then, the encoder maps out two vectors through a series of neural network layers (usually including fully connected layers and activation function layers), namely the mean vector (μ, indicating the central position of the current sample distribution in the latent space) and the standard deviation vector (σ, indicating the uncertainty of the sample in each dimension in the latent space).

[0064] In this step, after extracting the mean vector and standard deviation vector, the position value of each first traffic sample tensor unit can be subjected to the combined action of random perturbation and feature domain perturbation control items based on the mean vector and standard deviation vector to generate a corresponding feature representation vector in the latent space. The feature representation vector combines the central features and scale features of the position value of the normal network traffic packet sample tensor unit, and integrates the disturbance suppression capability determined by the feature domain disturbance control item.

[0065] Step 240: Input the feature representation vector corresponding to the first traffic sample in the latent space into the distribution-guided decoder network, and reconstruct the network traffic feature position value through the distribution-guided decoder network output.

[0066] In this step, the feature representation vector corresponding to the first traffic sample in the latent space can be input into the distribution-guided decoder network. The distribution-guided decoder network generates the corresponding reconstructed network traffic feature position value based on the feature representation vector. The reconstructed network traffic feature position value output by the distribution-guided decoder network is consistent with the normal network traffic distribution in terms of statistical characteristics, and suppresses the boundary disturbance component when restoring the position value of the tensor unit of the boundary attack network traffic packet sample.

[0067] Step 250: construct a loss function based on the reconstructed network traffic feature position value, and jointly train the conditional attention encoder network and the distribution-guided decoder network through the loss function to obtain a latent space prior probability model.

[0068] In this step, a loss function corresponding to the variational autoencoder can be constructed based on the reconstructed network traffic feature position value. Specifically, the loss function includes a reconstruction error term of the boundary attack network traffic feature position value, a latent space feature domain smoothness constraint term, and a Kullback Leibler divergenc (KL divergence) regularization term.

[0069] Among them, the reconstruction error term is used to measure the distance between the position value of the reconstructed network traffic feature and the position value of the original network traffic capture tensor unit; the latent space feature domain smoothness constraint term is used to measure the disturbance suppression ability of the feature representation vector; the KL divergence regularization term is used to measure the distribution offset between the latent space distribution and the standard normal distribution.

[0070] In a specific embodiment, the conditional attention encoder network and the distribution-guided decoder network can be jointly trained end-to-end by minimizing the variational autoencoder loss function. The network parameters are updated during the training process to obtain a latent space prior probability model suitable for the boundary attack network traffic denoising task.

[0071] Step 260: Input the second traffic sample into the conditional attention encoder network, and encode the position value of each second traffic sample tensor unit through the conditional attention encoder network to obtain an abnormal traffic feature vector.

[0072] In this step, all second traffic samples can be input into the trained conditional attention encoder network, and the conditional attention encoder network can be used to encode the position value of each abnormal network traffic packet sample tensor unit to obtain an abnormal traffic feature vector consistent with the latent space prior probability model.

[0073] The advantage of this setting is that by constructing an improved variational autoencoder structure that combines conditional attention encoding with feature domain disturbance suppression, it is possible to adaptively allocate encoding weights to areas sensitive to boundary attack disturbances, and introduce feature domain disturbance control terms in the latent space modeling stage, thereby achieving dynamic filtering and suppression of abnormal high-frequency features, and accurately separating boundary disturbances from normal traffic patterns. This allows the reconstruction results to maintain the global structure of the traffic while effectively suppressing the propagation of adversarial noise in high-dimensional space, significantly improving the noise reduction accuracy and model adversarial robustness.

[0074] Step 270: Use the latent space prior probability model to decode the abnormal traffic feature vector to obtain a reconstructed network traffic tensor, input the reconstructed network traffic tensor and the original network traffic capture tensor into a preset tensor decomposition model, and output a low-rank reconstructed network traffic tensor through the tensor decomposition model.

[0075] In one implementation of this embodiment, step 270 may include the following steps:

[0076] Step 271: Input the abnormal traffic feature vector into the trained distribution-guided decoder network to generate a reconstructed network traffic tensor. ,in, 、 、 、 They represent the number of time windows, the number of protocol categories, the number of topological path combinations, and the number of content features, respectively;

[0077] Step 272: Reconstruct the network traffic tensor Capturing tensors with raw network traffic A common input distribution reconstruction tensor decomposition model is constructed to construct a low-rank tensor decomposition problem with distribution alignment constraints. While maintaining the integrity of the four-dimensional structure of the network traffic tensor, the high-frequency noise components introduced by the boundary attack perturbation are compressed into a low-rank space while restoring the normal statistical distribution shape.

[0078] Step 273: Set the low-rank tensor decomposition form of the reconstructed network traffic tensor to a Canonical Polyadic decomposition structure:

[0079] ;

[0080] in, represents the low-rank reconstructed network traffic tensor, is the tensor rank, For the The weight coefficient of the rank component, 、 、 、 Represent the component vectors in the time dimension, protocol dimension, topology dimension and content dimension respectively, Represents the outer product operation of vectors;

[0081] The above formula uses Canonical Polyadic decomposition to express the reconstructed network traffic tensor after noise reduction as the sum of the weighted outer products of a set of rank components in a multi-dimensional feature space. Any high-order network traffic tensor is expressed as the outer product of several groups of time dimension components, protocol dimension components, topology dimension components, and content dimension components. Each group of components corresponds to a rank component, and each rank component has its weight coefficient, which not only enables the traffic data to be structurally decomposed in each dimension, but also compresses high-dimensional noise, so that the residual high-frequency interference signal can be effectively diluted in the low-rank space.

[0082] Step 274: Construct a tensor decomposition loss function for the distribution reconstruction tensor decomposition , including the low-rank reconstruction network traffic tensor structure reconstruction error term and traffic statistical distribution consistency constraint term:

[0083] ;

[0084] in, represents the Frobenius norm, represents the KL divergence of the statistical distribution of the low-rank reconstructed network traffic tensor and the original network traffic capture tensor, is the distribution constraint weight parameter;

[0085] The core of the tensor decomposition loss function lies in the organic combination of structural reconstruction error and statistical distribution consistency constraints. Traditional tensor decomposition often only optimizes the structural reconstruction error, that is, using the Frobenius norm to minimize the difference between the low-rank reconstructed tensor and the input tensor. However, this embodiment introduces a KL divergence distribution alignment term into the tensor decomposition loss function, requiring the low-rank reconstructed tensor to maintain a consistent global distribution with the original network traffic capture tensor. This not only ensures the integrity of structural information, but also systematically addresses the problems of distribution drift and distribution distortion under adversarial perturbations.

[0086] Step 275: By minimizing the objective function , using the tensor decomposition model to jointly iteratively optimize the component vectors and weight coefficients in the time dimension, protocol dimension, topology dimension and content dimension , while keeping the tensor structure intact, the high-frequency components of the boundary attack disturbance are compressed and the original distribution characteristics are restored.

[0087] Step 280: Based on the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor, the variational autoencoder and the tensor decomposition model are jointly trained until the reconstruction error meets the preset convergence condition, and the denoised network traffic tensor is output.

[0088] In one implementation of this embodiment, based on the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor, the variational autoencoder and the tensor decomposition model are jointly trained until the reconstruction error meets the preset convergence condition and the denoised network traffic tensor is output, including:

[0089] Step 281: Determine the reconstruction error tensor of the corresponding position elements between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor;

[0090] In this step, the reconstruction error tensor is used to measure the numerical differences between the two at each position in the time dimension, protocol dimension, topology dimension and content dimension. Each position element in the reconstruction error tensor is obtained by subtracting the value of the position in the original network traffic capture tensor from the value of the same position in the low-rank reconstructed network traffic tensor.

[0091] Step 282: construct an error energy spectrum vector based on all element values ​​in the reconstructed error tensor, and determine a noise energy threshold based on the error energy spectrum vector;

[0092] In this step, the absolute values ​​of all elements in the reconstruction error tensor are taken and expanded into a one-dimensional vector to obtain the error energy spectrum vector. The error energy spectrum vector is used to uniformly represent the reconstruction error energy in all dimensions. The error energy spectrum vector is obtained by expanding the reconstruction error tensor element by element and taking the absolute value.

[0093] The noise energy threshold is used to measure the boundary between acceptable error and abnormal error. The noise energy threshold is obtained by calculating the quantile value at the specified quantile factor position in the error energy spectrum vector. The quantile factor controls the proportion of error energy retained.

[0094] Step 283: Construct a joint loss function corresponding to the variational autoencoder and the tensor decomposition model;

[0095] In this step, the joint loss function can be constructed , the joint loss function is composed of the variational autoencoder loss function , tensor decomposition loss function and noise constraint Weighted calculation.

[0096] Step 284: Iteratively train the variational autoencoder and the tensor decomposition model according to the joint loss function and the noise energy threshold until the reconstructed error tensor meets the preset convergence condition and outputs the denoised network traffic tensor.

[0097] In this step, by minimizing the joint loss function Optimize the parameters of the conditional attention encoder network and the distribution-guided decoder network in the variational autoencoder, as well as the weight coefficients of the component vectors and rank components in the time dimension, protocol dimension, topology dimension, and content dimension in the tensor decomposition model, so that the variational autoencoder and the tensor decomposition model form a synergistic mechanism in the denoising task. At the same time, return to execute steps 250-270, and dynamically adjust the noise energy threshold according to the current reconstruction error in each round of iteration until the low-rank reconstructed network traffic tensor reaches the minimum feasible reconstruction error and has the ability to suppress disturbances, and output the final low-rank reconstructed network traffic tensor as the denoising network traffic tensor.

[0098] In this embodiment, the noise constraint term is used to penalize all reconstruction error regions that are greater than the noise energy threshold. The contribution of each error term in the noise constraint term is determined by the square value of the reconstruction error at that position, and is included in the total only when the error value exceeds the noise energy threshold.

[0099] By proposing a joint adaptive threshold and collaborative reverse update mechanism, taking the global energy spectrum of the low-rank reconstruction error as the dynamic basis of the adaptive threshold, and constructing an end-to-end joint loss function, the variational autoencoder and tensor decomposition model parameters are closed-loop optimized in each round of iteration. This not only improves the adaptability and scene generalization ability of the noise reduction effect, but also ensures the real-time processing performance of the encoder and model in complex network environments and high-traffic and high-concurrency backgrounds.

[0100] Step 290: Generate a network traffic data security report based on the abnormal traffic feature vector, the low-rank reconstructed network traffic tensor, and the noise energy threshold; and push the network traffic data security report and the noise-reduced network traffic tensor to the monitoring platform.

[0101] In this step, specifically, the abnormal traffic feature vector, low-rank reconstructed network traffic tensor and noise energy threshold can be synchronously recorded in the security audit log to generate an explainable report containing traceability information, and then the explainable report and the noise reduction network traffic tensor can be synchronously pushed to the monitoring platform.

[0102] The technical solution provided by the embodiment of the present invention obtains the original network traffic capture tensor corresponding to the network interface, determines the boundary attack probability and the boundary attack discrimination result of the original network traffic capture tensor packet by packet through the boundary attack discrimination model, divides the original network traffic capture tensor into a first traffic sample that has not been subjected to a boundary attack and a second traffic sample that has been subjected to a boundary attack, and uses a variational autoencoder to extract the mean vector and the standard deviation vector corresponding to the first traffic sample; based on the mean vector and the standard deviation vector, generates a feature representation vector corresponding to each first traffic sample in the latent space, inputs the feature representation vector corresponding to the first traffic sample in the latent space into a distribution-guided decoder network, reconstructs the network traffic feature position value through the distribution-guided decoder network output, constructs a loss function based on the reconstructed network traffic feature position value, and jointly trains the conditional attention encoder network and the distribution-guided decoder network through the loss function to obtain The latent space prior probability model encodes the position value of each second traffic sample tensor unit through the conditional attention encoder network to obtain the abnormal traffic feature vector, and uses the latent space prior probability model to decode the abnormal traffic feature vector to obtain the reconstructed network traffic tensor. The reconstructed network traffic tensor and the original network traffic capture tensor are input into the tensor decomposition model, and the low-rank reconstructed network traffic tensor is output through the tensor decomposition model. According to the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor, the variational autoencoder and the tensor decomposition model are jointly trained until the reconstruction error meets the preset convergence condition, and the denoised network traffic tensor is output to generate a network traffic data security report. The technical means of pushing the network traffic data security report and the denoised network traffic tensor to the monitoring platform can effectively denoise the network traffic data and improve the overall level of network security situation awareness and governance.

[0103] Figure 3 This is a schematic diagram of the structure of a network traffic data noise reduction device provided by an embodiment of the present invention, which is applied to electronic devices such as Figure 3 As shown, the device includes: a traffic collection module 310, a sample processing module 320, a vector decoding module 330 and a joint training module 340.

[0104] The traffic collection module 310 is configured to obtain a raw network traffic capture tensor corresponding to the network interface, and divide the raw network traffic capture tensor into a first traffic sample that has not been subjected to a boundary attack, and a second traffic sample that has been subjected to a boundary attack;

[0105] The sample processing module 320 is configured to use the first traffic sample to train a variational autoencoder to obtain a latent space prior probability model, and use the variational autoencoder to encode the second traffic sample to obtain an abnormal traffic feature vector;

[0106] A vector decoding module 330 is configured to decode the abnormal traffic feature vector using a latent space prior probability model to obtain a reconstructed network traffic tensor, input the reconstructed network traffic tensor and the original network traffic capture tensor into a preset tensor decomposition model, and output a low-rank reconstructed network traffic tensor through the tensor decomposition model;

[0107] The joint training module 340 is used to jointly train the variational autoencoder and the tensor decomposition model according to the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor until the reconstruction error meets the preset convergence condition and output the denoised network traffic tensor.

[0108] The technical solution provided by the embodiment of the present invention obtains the original network traffic capture tensor corresponding to the network interface, divides the original network traffic capture tensor into a first traffic sample that has not been attacked by a boundary, and a second traffic sample that has been attacked by a boundary, uses the first traffic sample to train the variational autoencoder to obtain a latent space prior probability model, and uses the variational autoencoder to encode the second traffic sample to obtain an abnormal traffic feature vector, uses the latent space prior probability model to decode the abnormal traffic feature vector to obtain a reconstructed network traffic tensor, inputs the reconstructed network traffic tensor and the original network traffic capture tensor into a preset tensor decomposition model, outputs a low-rank reconstructed network traffic tensor through the tensor decomposition model, and jointly trains the variational autoencoder and the tensor decomposition model according to the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor until the reconstruction error meets the preset convergence condition, and outputs the denoised network traffic tensor. The technical means can effectively denoise network traffic data and improve the overall level of network security situation awareness and governance.

[0109] Based on the above embodiment, the variational autoencoder consists of a conditional attention encoder network and a distribution guided decoder network.

[0110] The traffic collection module 310 includes:

[0111] a data partitioning unit, configured to determine a boundary attack probability for each packet of the original network traffic capture tensor using a preset boundary attack discrimination model, and output a boundary attack discrimination result corresponding to the original network traffic capture tensor based on the boundary attack probability; and, based on the boundary attack discrimination result, partition the original network traffic capture tensor into a first traffic sample that has not been subjected to a boundary attack and a second traffic sample that has been subjected to a boundary attack;

[0112] The data processing unit is used to collect the traffic packet data corresponding to the network interface in real time; format the timestamp dimension, protocol stack dimension, topology dimension and content dimension features contained in the traffic packet data to obtain the original network traffic capture tensor; and construct the position value of each tensor unit in the original network traffic capture tensor based on the discretized time window, protocol category set, topology path set and content feature set.

[0113] The sample processing module 320 includes:

[0114] A vector extraction unit, configured to extract a mean vector and a standard deviation vector corresponding to the first traffic sample using a variational autoencoder; and generate a feature representation vector corresponding to each first traffic sample in a latent space based on the mean vector and the standard deviation vector;

[0115] a variational autoencoder training unit, configured to input a feature representation vector corresponding to the first traffic sample in the latent space into a distribution-guided decoder network, and output a reconstructed network traffic feature position value through the distribution-guided decoder network; construct a loss function based on the reconstructed network traffic feature position value, and jointly train the conditional attention encoder network and the distribution-guided decoder network using the loss function to obtain a latent space prior probability model;

[0116] The sample encoding unit is used to input the second traffic sample into the conditional attention encoder network, encode the position value of each second traffic sample tensor unit through the conditional attention encoder network, and obtain the abnormal traffic feature vector.

[0117] The joint training module 340 includes:

[0118] A reconstruction error determination unit is used to determine a reconstruction error tensor of elements at corresponding positions between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor; construct an error energy spectrum vector based on all element values ​​in the reconstruction error tensor, and determine a noise energy threshold based on the error energy spectrum vector;

[0119] An iterative training unit is used to construct a joint loss function corresponding to the variational autoencoder and the tensor decomposition model; according to the joint loss function and the noise energy threshold, the variational autoencoder and the tensor decomposition model are iteratively trained until the reconstructed error tensor meets the preset convergence condition and the denoised network traffic tensor is output.

[0120] The device further comprises:

[0121] The security report generation module is used to generate a network traffic data security report based on the abnormal traffic feature vector, the low-rank reconstructed network traffic tensor and the noise energy threshold; and push the network traffic data security report and the noise-reduced network traffic tensor to the monitoring platform.

[0122] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not fully described in the embodiments of the present invention, please refer to the methods provided by all the above embodiments of the present invention.

[0123] Figure 4 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0124] like Figure 4 As shown, electronic device 10 includes at least one processor 11 and memory, such as read-only memory (ROM) 12 and random access memory (RAM) 13, communicatively connected to at least one processor 11. The memory stores computer programs executable by the at least one processor. Processor 11 can perform various appropriate actions and processes based on the computer programs stored in ROM 12 or loaded from storage unit 18 into RAM 13. RAM 13 can also store various programs and data required for the operation of electronic device 10. Processor 11, ROM 12, and RAM 13 are interconnected via bus 14. An input / output (I / O) interface 15 is also connected to bus 14.

[0125] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0126] Processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any other suitable processor, controller, microcontroller, etc. Processor 11 executes the various methods and processes described above, such as the network traffic data denoising method.

[0127] In some embodiments, the network traffic data denoising method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the network traffic data denoising method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the network traffic data denoising method in any other suitable manner (e.g., via firmware).

[0128] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0130] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or apparatus. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device that has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0133] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within a cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0134] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0135] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for reducing noise in network traffic data, characterized in that: The method comprises: Obtaining an original network traffic capture tensor corresponding to the network interface, and dividing the original network traffic capture tensor into a first traffic sample that has not been attacked by the boundary, and a second traffic sample that has been attacked by the boundary; The first traffic sample is used to train the variational autoencoder to obtain a latent space prior probability model, and the second traffic sample is encoded using the variational autoencoder to obtain an abnormal traffic feature vector; Using a latent space prior probability model to decode the abnormal traffic feature vector to obtain a reconstructed network traffic tensor, inputting the reconstructed network traffic tensor and the original network traffic capture tensor into a preset tensor decomposition model, and outputting a low-rank reconstructed network traffic tensor through the tensor decomposition model; According to the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic captured tensor, the variational autoencoder and the tensor decomposition model are jointly trained until the reconstruction error meets the preset convergence condition and the denoised network traffic tensor is output.

2. The method according to claim 1, characterized in that Dividing the original network traffic capture tensor into a first traffic sample that has not been attacked by a boundary and a second traffic sample that has been attacked by a boundary, including: By using a preset boundary attack discrimination model, the boundary attack probability of the original network traffic capture tensor is determined packet by packet, and the boundary attack discrimination result corresponding to the original network traffic capture tensor is output according to the boundary attack probability; According to the boundary attack identification result, the original network traffic capture tensor is divided into a first traffic sample that has not been subjected to the boundary attack and a second traffic sample that has been subjected to the boundary attack.

3. The method according to claim 1, characterized in that The variational autoencoder consists of a conditional attention encoder network and a distribution-guided decoder network; The first traffic sample is used to train the variational autoencoder to obtain a latent space prior probability model, including: Use a variational autoencoder to extract the mean vector and standard deviation vector corresponding to the first traffic sample; Generating a feature representation vector corresponding to each first flow sample in the latent space according to the mean vector and the standard deviation vector; Input the feature representation vector corresponding to the first traffic sample in the latent space into the distribution-guided decoder network, and reconstruct the network traffic feature position value through the distribution-guided decoder network output; A loss function is constructed according to the position value of the reconstructed network traffic feature, and the conditional attention encoder network and the distribution-guided decoder network are jointly trained using the loss function to obtain a latent space prior probability model.

4. The method according to claim 3, characterized in that The second traffic sample is encoded using a variational autoencoder to obtain an abnormal traffic feature vector, including: The second traffic sample is input into the conditional attention encoder network, and the position value of each second traffic sample tensor unit is encoded by the conditional attention encoder network to obtain the abnormal traffic feature vector.

5. The method according to claim 1, wherein Based on the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor, the variational autoencoder and the tensor decomposition model are jointly trained until the reconstruction error meets the preset convergence condition and the denoised network traffic tensor is output, including: Determine a reconstruction error tensor of corresponding position elements between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor; According to all element values ​​in the reconstructed error tensor, an error energy spectrum vector is constructed, and a noise energy threshold is determined according to the error energy spectrum vector; Construct a joint loss function corresponding to the variational autoencoder and the tensor decomposition model; According to the joint loss function and the noise energy threshold, the variational autoencoder and the tensor decomposition model are iteratively trained until the reconstructed error tensor meets the preset convergence condition and the denoised network traffic tensor is output.

6. The method according to claim 1, characterized in that Get the raw network traffic capture tensor corresponding to the network interface, including: Collect traffic packet data corresponding to the network interface in real time; Formatting the timestamp dimension, protocol stack dimension, topology dimension, and content dimension features contained in the traffic packet data to obtain an original network traffic capture tensor; According to the discretized time window, protocol category set, topology path set and content feature set, the position value of each tensor unit in the original network traffic capture tensor is constructed.

7. The method according to claim 5, characterized in that After outputting the denoising network traffic tensor, also include: Generate a network traffic data security report based on the abnormal traffic feature vector, low-rank reconstructed network traffic tensor, and noise energy threshold; The network traffic data security report and the noise-reduced network traffic tensor are pushed to the monitoring platform.

8. A network traffic data noise reduction device, characterized in that: The device comprises: a traffic collection module, configured to obtain a raw network traffic capture tensor corresponding to a network interface, and divide the raw network traffic capture tensor into a first traffic sample that has not been attacked by a boundary, and a second traffic sample that has been attacked by a boundary; A sample processing module is used to train a variational autoencoder using the first traffic sample to obtain a latent space prior probability model, and to encode the second traffic sample using the variational autoencoder to obtain an abnormal traffic feature vector; A vector decoding module is used to decode the abnormal traffic feature vector using a latent space prior probability model to obtain a reconstructed network traffic tensor, input the reconstructed network traffic tensor and the original network traffic capture tensor into a preset tensor decomposition model, and output a low-rank reconstructed network traffic tensor through the tensor decomposition model; The joint training module is used to jointly train the variational autoencoder and the tensor decomposition model according to the reconstruction error between the low-rank reconstructed network traffic tensor and the original network traffic capture tensor until the reconstruction error meets the preset convergence condition and outputs the denoised network traffic tensor.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the network traffic data denoising method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the network traffic data denoising method according to any one of claims 1 to 7 when executed.

Images