A Network Anomaly Detection Method Based on Variational Autoencoder

Abstract

This invention provides a network anomaly detection method based on variational autoencoders, comprising the following steps: S1: network data acquisition; S2: data preprocessing; S3: construction of a variational autoencoder model integrating multiple algorithms; S4: model training and optimization integrating multiple algorithms; S5: determination of anomaly detection threshold; S6: network anomaly detection inference integrating multiple algorithms; S7: anomaly result verification and model feedback optimization; S8: real-time network anomaly detection deployment. This invention improves the quality of network data feature extraction and the modeling accuracy of potential distributions through multi-algorithm collaborative modeling and bidirectional interactive optimization, enhances the model's adaptability to distribution shifts and its ability to identify complex anomaly behaviors, ultimately achieving accurate, efficient, and robust network anomaly detection, adapting to dynamically changing network environments, and providing reliable technical support for network security protection.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of network security detection and machine learning applications, and in particular to a network anomaly detection method based on variational autoencoders. Background Technology

[0002] With the rapid development of network technology, the network environment is becoming increasingly complex. Malicious network behaviors such as DDoS (Distributed Denial of Service) attacks, port scanning, login cracking, and data breaches are occurring frequently, posing a serious threat to network security. Network anomaly detection has become a core component of ensuring network security. Existing network anomaly detection technologies are mainly divided into two categories: traditional rule-based detection and machine learning-based detection.

[0003] Traditional rule-based detection relies on manually preset security rules to match abnormal behavior. Although it is simple to implement, it is not adaptable to new attack behaviors, has high rule maintenance costs, and is prone to missed detections.

[0004] Machine learning-based detection methods, such as traditional variational autoencoders (VAEs) and support vector machines (SVMs), employ data-driven modeling and possess a certain generalization ability. However, they have several limitations: First, traditional VAEs assume that the latent space follows a simple Gaussian distribution, which is difficult to adapt to the non-Gaussian distribution characteristics of complex network data, resulting in insufficient latent feature representation. Second, the local neighborhood structure of the data is easily lost during feature dimensionality reduction, affecting subsequent detection accuracy. Third, the distribution offset between the training and test sets is not considered, and detection performance significantly decreases when the data distribution changes dynamically in the actual network environment. Fourth, the posterior estimation of the latent distribution is not accurate enough, and the various algorithm modules are mostly unidirectional, lacking a collaborative optimization mechanism, making it difficult to fully leverage the complementary advantages of the algorithms.

[0005] The aforementioned shortcomings make it difficult for existing technologies to meet the security protection requirements of complex network environments in terms of detection accuracy, robustness, and dynamic adaptability. Summary of the Invention

[0006] This invention provides a network anomaly detection method based on variational autoencoders, overcoming the shortcomings of existing network anomaly detection technologies, such as insufficient latent feature representation, loss of local structure, poor distribution adaptability, and weak algorithm synergy. Through multi-algorithm collaborative modeling and bidirectional interactive optimization, it improves the extraction quality of network data features and the modeling accuracy of latent distributions, enhances the model's adaptability to distribution shifts and its ability to identify complex abnormal behaviors, and ultimately achieves accurate, efficient, and robust network anomaly detection, adapting to dynamically changing network environments and providing reliable technical support for network security protection.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A network anomaly detection method based on variational autoencoders includes the following steps:

[0009] S1: Collect network traffic data and network log data in the target network environment to form raw network data;

[0010] S2: Preprocess the raw network data and output standardized structured feature data;

[0011] S3: Construct a variational autoencoder model that integrates multiple algorithms. Dimensionality-reduced features are obtained by performing dimensionality reduction on standardized structured feature data through local preservation projection, and input into the encoder to generate an initial latent vector. This vector is then transformed using an inverse autoregressive flow to obtain the transformed latent vector. A sparsified posterior estimate is performed on the transformed latent vector using sparse Gaussian process variational inference. The decoder reconstructs the features based on the sparsified posterior estimate. The reconstruction loss is calculated based on the standardized structured feature data and the reconstructed features. The KL divergence loss is calculated based on the prior and posterior distributions of the sparse Gaussian process variational inference. The kernel mean matching loss is calculated based on the dimensionality-reduced features. The reconstruction loss, KL divergence loss, and kernel mean matching loss are then fused to construct the total loss function.

[0012] S4: Train a variational autoencoder model that integrates multiple algorithms using training set data from standardized structured feature data. Iteratively optimize and minimize the total loss function, update the relevant parameters of local preserving projection, inverse autoregressive flow, sparse Gaussian process variational inference and kernel mean matching loss until the model converges and output the trained model.

[0013] S5: Input the validation set data from the standardized structured feature data into the trained model, calculate the anomaly detection error, and determine the anomaly detection threshold based on the distribution characteristics of the anomaly detection error;

[0014] S6: Preprocess the real-time collected network data to obtain real-time standardized structured feature data, input it into the trained model, calculate the real-time final anomaly judgment error, compare it with the anomaly detection threshold, and output the anomaly detection result.

[0015] In this specification, the network anomaly detection method based on variational autoencoder also includes S7: manually verifying the anomaly detection results and labeling them with real labels, adding the labeled samples to the original training data, re-executing steps S2, S4 and S5, updating the model parameters and anomaly detection threshold, and obtaining the optimized model and the updated anomaly detection threshold.

[0016] S8: Deploy the optimized model and updated anomaly detection threshold to the real-time detection system to achieve real-time anomaly monitoring of the target network.

[0017] In this specification, there is a two-way interaction between local preservation projection and sparse Gaussian process variational inference in S3. Specifically, sparse Gaussian process variational inference updates the neighborhood weight matrix of local preservation projection through its inter-sample kernel matrix. Local preservation projection recalculates the projection matrix based on the updated neighborhood weight matrix, thereby correcting the dimensionality reduction features.

[0018] In this specification, there is a parameter feedback adjustment process between the inverse autoregressive flow and the sparse Gaussian process variational inference in S3. Specifically, the initial value of the transformation parameter of the inverse autoregressive flow is randomly initialized, and during the training process, it is adjusted by the posterior mean output by the sparse Gaussian process variational inference, so that the transformation of the inverse autoregressive flow fits the potential distribution characteristics modeled by the sparse Gaussian process variational inference.

[0019] In this specification, the calculation of kernel mean matching loss in S3 is based on the dimensionality reduction features of locally preserving projection output. By introducing training set sample weights, the kernel mean of the dimensionality reduction features of the training set and the test set tends to be consistent in the regenerated kernel Hilbert space, thus achieving distribution alignment.

[0020] In this specification, the iterative optimization process for training the variational autoencoder model that integrates multiple algorithms in S4 is as follows: in each round of training, the projection matrix of the local preserving projection, the transformation parameters of the inverse autoregressive flow, the induction point and kernel parameters of the sparse Gaussian process variational inference, and the sample weights of the kernel mean matching loss are updated sequentially, and then all learnable parameters are updated synchronously based on the total loss function.

[0021] In this specification, the convergence criteria for model S4 are: reconstruction loss is less than the preset reconstruction loss threshold, KL divergence loss is less than the preset KL divergence loss threshold, kernel mean matching loss is less than the preset kernel mean matching loss threshold, and the total loss has not decreased for a consecutive preset number of rounds.

[0022] In this specification, the determination of the anomaly detection threshold in S5 adopts a progressive optimization strategy. Specifically, the initial threshold is first determined based on the quantiles of the final anomaly judgment error of all samples in the validation set. Then, the mean and standard deviation of the final anomaly judgment error of normal samples in the validation set are calculated through a sliding window, and the anomaly detection threshold is dynamically updated.

[0023] In this specification, the real-time final anomaly determination error in S6 is a weighted sum of the basic reconstruction error and the kernel mean matching distribution correction term. The basic reconstruction error is calculated based on the real-time standardized structured feature data and the corresponding reconstruction features, and the kernel mean matching distribution correction term is calculated based on the distribution difference between the real-time dimensionality reduction features and the training set dimensionality reduction features.

[0024] In this manual, the anomaly detection results in S6 are categorized into four levels: normal, mild anomaly, moderate anomaly, and severe anomaly, based on the magnitude of the real-time final anomaly judgment error exceeding the anomaly detection threshold.

[0025] In summary, the present invention has at least the following beneficial effects:

[0026] 1. Enhance local feature preservation capability: By using the local preserving projection algorithm to perform local shape-preserving dimensionality reduction on the input features, the computational complexity is reduced while the local neighborhood structure of the network data is preserved to the maximum extent, providing a high-quality feature foundation for subsequent accurate modeling and reducing the problem of missed detections caused by feature loss;

[0027] 2. Enhance the latent space representation capability: The inverse autoregressive flow algorithm is integrated to perform an invertible transformation on the latent vector of the traditional variational autoencoder, breaking the limitation of the Gaussian distribution assumption, adapting to the non-Gaussian distribution characteristics of complex network data, improving the discriminativeness of latent features, and helping to accurately identify hidden abnormal behaviors.

[0028] 3. Optimize the accuracy and sparsity of posterior estimation: Introduce the sparse Gaussian process variational inference algorithm to perform sparsified posterior estimation of the latent distribution after inverse autoregressive flow transformation, replacing the simple Gaussian posterior assumption of the traditional variational autoencoder. At the same time, through bidirectional interactive updates with local preserving projection, the feature dimensionality reduction and latent modeling effects are further optimized.

[0029] 4. Improve the robustness of the model: Incorporate kernel mean matching loss to align the feature distributions of the training and test sets, effectively mitigating the impact of data distribution shifts on detection performance in real-world network environments and enhancing the model's adaptability in dynamic network environments;

[0030] 5. Achieve multi-algorithm collaborative optimization: Construct a two-way interaction mechanism between local preservation projection and sparse Gaussian process variational inference, and between inverse autoregressive flow and sparse Gaussian process variational inference. Integrate the loss functions of multiple algorithms to form a total optimization objective, give full play to the complementary advantages of each algorithm, and improve the overall detection accuracy and reliability.

[0031] 6. Adapt to real-time detection requirements: By optimizing the model structure and inference process, low-latency data preprocessing and inference calculations are achieved. Combined with data feedback and closed-loop optimization mechanisms, the model can continuously adapt to new network attack behaviors, ensuring the long-term effectiveness of the detection system. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic flowchart of the network anomaly detection method based on variational autoencoder involved in this invention.

[0034] Figure 2 This is a schematic diagram of the data processing flow involved in this invention.

[0035] Figure 3 This is a schematic diagram illustrating the process of constructing the fusion model involved in this invention.

[0036] Figure 4 This is a schematic diagram of the real-time detection deployment process involved in this invention. Detailed Implementation

[0037] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0038] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0039] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0040] like Figure 1 As shown, this embodiment provides a network anomaly detection method based on a variational autoencoder, including:

[0041] S1: Network Data Acquisition

[0042] To ensure that subsequent model training and inference can obtain comprehensive and high-quality input data, this step carries out multi-dimensional and real-time data collection for the target network environment, covering two core data types: network traffic and network logs. The collection process strictly follows the principles of full-dimensional coverage, low-latency capture, and traceable storage.

[0043] The deployment of data sources covers network transmission links and core device nodes: By deploying traffic capture tools such as Wireshark and tcpdump on the mirror ports of network gateways and core switches, all network packets passing through the links are captured in real time; system logs of network devices such as routers and firewalls are read remotely via the SSH protocol; server operating system logs are collected via Windows Event Viewer and Linux syslog service; and application layer logs such as web services and database services are collected via application service log interfaces, ensuring no data collection blind spots.

[0044] The collected data parameters are precisely matched to the subsequent feature extraction requirements; network traffic data includes: packet capture timestamp (accurate to milliseconds), source IP address, destination IP address, source port number, destination port number, transmission protocol type, packet length, packet transmission rate, and TCP connection status (established, closed, timeout, half-closed, etc.); network log data must include log generation timestamp (accurate to milliseconds), event type, operation subject (username, process ID, device identifier), operation object (file path, port number, database table name, API interface address), and event result (success, failure, exception, unknown).

[0045] The data collection method combines real-time streaming acquisition with periodic storage: traffic data is captured every millisecond to ensure no sudden traffic spikes are missed; log data is collected using a real-time monitoring + batch reading mode, scanning the log file directory every 500 milliseconds to read newly added log entries. All collected data is stored in a distributed database, HBase, in JSON format. Each data entry is uniquely identified by a combination of "timestamp + source IP address + destination IP address + core event type," and metadata such as the data acquisition device ID and acquisition timestamp are also recorded.

[0046] S2: Data Preprocessing

[0047] This step standardizes the raw network data collected by S1. Through a complete process of data cleaning, feature extraction, and data standardization, the unstructured and heterogeneous raw data is transformed into structured feature vectors. The data processing flow is as follows: Figure 2 As shown.

[0048] The data cleaning phase focuses on addressing missing, outlier, and duplicate issues in the original data: For numerical data such as packet length and transmission rate, the median of this field in the entire dataset is used to fill in missing values, avoiding the influence of outliers on mean filling; for categorical data such as protocol type and event type, the mode of this field is used to fill in missing values, ensuring that the filled data conforms to the actual distribution pattern; if the missing field percentage of a certain data entry exceeds 30%, that data entry is directly removed to avoid invalid data affecting model performance. Outlier handling uses the interquartile range (IQR) method. For numerical parameters such as packet length and transmission rate, the first quartile Q1 and the third quartile Q3 are calculated, determining the outlier judgment interval as [Q1-1.5×IQR, Q3+1.5×IQR]. Data exceeding this interval is marked as outliers and directly removed, ensuring the rationality of data distribution. Duplicate value handling uses "timestamp + source IP + destination IP + core event" as a unique identifier, traversing the entire dataset to delete duplicate entries, ensuring the uniqueness of each data entry.

[0049] The feature extraction stage constructs a multi-dimensional feature vector based on the semantic and statistical characteristics of the original data: Time feature extraction parses three dimensions from the timestamp: hour (0-23), day of the week (1-7), and whether it is a holiday (0=no, 1=yes). Holiday information is obtained by calling the public holiday API and stored in the local cache in advance to ensure feature extraction efficiency. Statistical feature extraction uses a 5-minute time window to perform sliding window statistics on network traffic data from the same source IP, calculating the mean, variance, maximum, and minimum of packet length, the mean and peak of transmission rate, and the number of TCP connection establishments, closures, and timeouts, forming an 8-dimensional statistical feature vector. Protocol and event feature extraction performs one-hot encoding on all protocol types and event types. The protocol types include five core protocols: TCP, UDP, ICMP, HTTP, and HTTPS, which are encoded into 5 dimensions. The event types include five core events: login success, login failure, file access, port connection, and process startup, which are encoded into 5 dimensions. Finally, the 3-dimensional time feature vector, the 8-dimensional statistical feature vector, and the 5-dimensional protocol and event feature vector are concatenated in sequence to form the original feature vector with dimension D=16.

[0050] The data standardization phase employs the Z-score standardization method to eliminate dimensional differences between features of different dimensions: for each feature dimension in the original feature vector, its mean μ and standard deviation σ on the training set are calculated according to the formula... Standardization processes are performed, among which These are the original eigenvalues. These are the standardized feature values. The standardized feature vectors follow a normal distribution with a mean of 0 and a variance of 1 in all dimensions. This avoids model training bias caused by differences in dimensions and is compatible with the kernel function modeling requirements of Gaussian kernel calculation and variational inference of sparse Gaussian processes in the local preservation projection module of S3. The final output is standardized structured feature data.

[0051] S3: Construction of Variational Autoencoder (VAE) Models Integrating Multiple Algorithms

[0052] This step breaks through the limitations of traditional variational autoencoders that rely solely on the Gaussian distribution assumption and simple encoding / decoding structures. It innovatively integrates four algorithms: Locality-Preserving Projections (LPP), Sparse Gaussian Process Variational Inference (SGPVI), Inverse Autoregressive Flow (IAF), and Kernel Mean Matching Loss (KMMLoss), to construct a composite model with the capabilities of local feature preservation, latent space enhancement, sparsified posterior estimation, and distribution alignment. The core collaborative logic is as follows: Local shape-preserving projection is used to achieve local dimensionality reduction of input features, providing a low-dimensional and structurally complete feature foundation for subsequent encoding; after the encoder outputs the initial latent vector, an inverse autoregressive flow is used for reversible transformation to break the Gaussian distribution constraint and enhance the expressive power of the latent space; sparse Gaussian process variational inference performs sparsified posterior estimation on the latent vector after the inverse autoregressive flow transformation, and its kernel matrix updates the neighborhood weights of the local shape-preserving projection, forming a bidirectional interaction; finally, the reconstruction loss, the KL divergence loss of sparse Gaussian process variational inference, and the kernel mean matching loss are fused to construct a total loss function that balances reconstruction accuracy, latent distribution rationality, and dissimilar robustness. The fusion model construction process is as follows: Figure 3 As shown.

[0053] 3.1 Feature Local Conformal Dimensionality Reduction Based on LPP

[0054] The core function of LPP is to reduce the dimensionality of the standardized structured feature vector output by S2. This reduces the computational complexity of subsequent encoding processes while preserving the local neighborhood structure of the data to the greatest extent possible, and provides more representative low-dimensional features for subsequent algorithms. The input to this step is the standardized feature vector output by S2. Its dimensions are It is composed of 3-dimensional temporal features, 8-dimensional statistical features, and 5-dimensional discrete features, and the output is a dimensionality-reduced feature vector. Dimensions are set as This dimension can achieve the optimal balance between local feature retention rate and computational efficiency.

[0055] First, a standardized feature matrix is ​​constructed for all samples. ,in The total number of samples, The dimension is To characterize the local neighborhood relationships between samples, a neighborhood weight matrix needs to be constructed. The matrix has dimensions of Its elements Indicates sample and The strength of neighborhood associations between samples. Specifically, the construction method is as follows: for each sample... ,use The nearest neighbor algorithm selects its neighborhood sample set. , A value of 10 can balance the cost of local structure capture and computation. belong or belong ,but The Gaussian kernel function is used for calculation, and the formula is as follows: ,in The Gaussian kernel bandwidth is given by a value of [value to be filled in]. That is, 0.5 times the mean of the Euclidean distance between neighboring samples, to ensure the smoothness of the change in neighborhood weights with distance; if Not belonging to The neighborhood and Not belonging to The neighborhood of, then .

[0056] Based on neighborhood weight matrix Further construct the Laplace matrix of LPP ,in It is a diagonal matrix, and its diagonal elements ,Right now No. The sum of the row elements is used to normalize the Laplace matrix to avoid numerical overflow.

[0057] The core objective of LPP is to solve the projection matrix. , dimension , so that the characteristic matrix The dimensionality-reduced feature matrix obtained after projection (dimension is) This can preserve the local neighborhood structure of the original data to the greatest extent possible. This objective is transformed into an optimization problem: ,in This represents the matrix trace operation, with the following constraints: , for A 3D identity matrix is ​​used to avoid trivial solutions to the projection matrix.

[0058] This optimization problem is equivalent to solving the generalized eigenvalue equation. ,in Let be the eigenvalues. After solving the equation, select the smallest eigenvalue. The eigenvectors corresponding to the eigenvalues ​​form the projection matrix. The feature matrix after dimensionality reduction at this time It can minimize the local neighborhood structure loss. For a single sample Its dimensionality reduction characteristics are .

[0059] 3.2 Encoder Design Integrating Inverse Autoregressive Flow

[0060] The core function of the encoder is to reduce the dimensionality of the feature vector after LPP. The latent vectors are mapped to the latent space to generate representative latent vectors. To overcome the limitation of the assumption that latent vectors in traditional VAEs follow a simple Gaussian distribution and to enhance the ability of the latent space to represent the features of complex network data, this scheme introduces a multi-layer IAF after the encoder for reversible transformation. At the same time, the parameters of the IAF are adjusted by the posterior distribution of SGPVI, forming a co-optimization between the encoder and SGPVI.

[0061] The encoder input is First, feature extraction is performed using two fully connected hidden layers. The calculation process of the first hidden layer is as follows: ,in The weight matrix of the first hidden layer has dimensions [dimensions missing]. ; For the corresponding bias vector, dimension The ReLU activation function is used to introduce non-linear feature representation; the calculation process of the second hidden layer is as follows: ,in This is the weight matrix for the second hidden layer, with dimensions [not specified]. ; For the corresponding bias vector, dimension Output For dimension The hidden feature vector.

[0062] Based on hidden feature vectors The encoder outputs the parameters of the initial latent distribution: the mean vector. With log-variance vector The calculation process is as follows: and ,in and This is the weight matrix, with all dimensions being 1. , As a potential spatial dimension; and The bias vector has dimensions of 1. Based on the reparameterization technique of variational autoencoders, an initial latent vector is generated. ,in For random noise that follows a standard normal distribution, , for A 3D identity matrix is ​​used to avoid the vanishing gradient problem.

[0063] To enhance the expressive power of the latent space, we introduce... The layers of IAF undergo cascaded invertible transformations. The transformation function for each layer of IAF is: , , No. The output latent vector of layer IAF ,in For the first The output of layer IAF, For the initial potential vector, For the first The parameter set of each layer of IAF. Each layer of IAF employs an autoregressive structure to ensure the invertibility of the transformation, and its first... Dimensional output The calculation process is as follows ;in and It is an autoregressive neural network, with the input being the previous... 1D vector, output as scalar, parameters are contained in This structure ensures the invertibility of the transformation, and the logarithmic determinant of the Jacobian matrix can be calculated efficiently, providing a foundation for subsequent probability modeling.

[0064] The log-likelihood correction term of the IAF transform is ,in For the The final latent vector after the layer IAF transformation is directly input into SGPVI for further processing.

[0065] To achieve coordinated optimization between IAF and SGPVI, the parameters of IAF... The adjustment is based on the feedback mean of the posterior distribution of SGPVI, and the adjustment formula is as follows: ,in For the first Initial values ​​of the parameters of layer IAF, The adjustment coefficient controls the degree of influence of SGPVI on the IAF parameter. The first output of SGPVI The adjustment mechanism, which uses the posterior mean of each subspace, makes the transformation of IAF more closely match the latent distribution characteristics modeled by SGPVI.

[0066] 3.3 Latent Distribution Modeling Based on SGPVI

[0067] The core function of SGPVI is to process the latent vector after IAF transformation. Sparse posterior estimation is performed to replace the simple Gaussian posterior assumption of traditional VAE, which improves the accuracy and sparsity of latent distribution modeling. At the same time, its kernel matrix feeds back to update the neighborhood weight matrix of LPP, forming a bidirectional interaction between SGPVI and LPP, which further optimizes the feature dimensionality reduction effect.

[0068] First, construct the latent matrix after the IAF transformation. , dimension To achieve sparse reasoning, we select... The induction points form an induction point matrix. , dimension , Experiments have shown that a balance can be achieved between sparsity and modeling accuracy. The induced points are obtained from the K-means algorithm... The selection is made from a pool of samples to ensure that the samples represent the core distribution characteristics of the potential space.

[0069] SGPVI uses radial basis functions (RBF) as its kernel function. The calculation process is as follows ,in The kernel bandwidth parameter is initially set to 1.0 and adaptively optimized during training. Based on the kernel function, three types of kernel matrices are constructed: the inter-sample kernel matrix. Dimension Sample-Induced Point Kernel Matrix (dimension) ), induced inter-point kernel matrix Dimension .

[0070] The variational posterior distribution of SGPVI is ,in It is a latent function in the latent space, which characterizes the mapping relationship between latent vectors and network data features; The posterior mean is... Let be the posterior covariance. The calculation process for the posterior mean is as follows: ,in Let be the mean vector of the induction points; the calculation process for the posterior covariance is as follows: ,in This represents the noise variance, used to avoid overfitting. for 3D identity matrix.

[0071] The objective of SGPVI is to minimize the KL divergence between the variational posterior distribution and the prior distribution, where the prior distribution is set as... The formula for calculating the KL divergence is: The KL divergence (denoted as ) It replaces the KL divergence of traditional VAEs and incorporates it into the total loss function.

[0072] To achieve bidirectional interaction between SGPVI and LPP, the inter-sample kernel matrix of SGPVI is first utilized. Update the neighborhood weight matrix of LPP The updated formula is as follows ,in For the Hadamard product, multiply element by element; For iteration rounds, for The maximum value in the matrix is ​​used to normalize the kernel matrix to... The interval ensures that it is consistent with... The weight ranges are consistent; subsequently, based on the updated... Recalculate the Laplace matrix of LPP The projection matrix is ​​updated by solving for generalized eigenvalues. This leads to the correction of dimensionality reduction features. This forms a closed-loop interaction: LPP dimensionality reduction → encoder → IAF → SGPVI → LPP update.

[0073] 3.4 Decoder Design

[0074] The core function of the decoder is to map the latent vectors processed by SGPVI back to the original feature space, generating reconstructed feature vectors and providing a basis for calculating the reconstruction loss. The decoder's input is the latent vector fused with SGPVI posterior covariance correction, ensuring that the reconstruction process can utilize the posterior information of SGPVI to improve reconstruction accuracy.

[0075] Decoder input vector The latent vector output by IAF The calculation process involves fusing the SGPVI posterior covariance with the calculated variance. ,in For noise vectors that follow a standard normal distribution ( ), For the posterior covariance matrix The square root of the vector consisting of the diagonal elements is used to adjust the intensity of the noise so that the decoder input can reflect the uncertainty of the underlying distribution.

[0076] The decoder employs a two-layer fully connected hidden layer structure symmetrical to the encoder. The calculation process of the first hidden layer is as follows: ,in The weight matrix has dimensions [missing information]. , For the bias vector, dimension The calculation process for the second hidden layer is as follows: ,in The weight matrix has dimensions [missing information]. , For the bias vector, dimension .

[0077] The decoder's output layer is a fully connected layer, outputting a reconstructed feature vector with the same dimension as the LPP dimensionality reduction feature. The calculation process is as follows ,in The weight matrix has dimensions [missing information]. , For the bias vector, dimension The sigmoid activation function is used to map the output value to... The interval is consistent with the range of the standardized input features.

[0078] To calculate the reconstruction loss, the dimensionality reduction features of the reconstruction need to be... The process of restoring to the original feature dimensions is as follows: ,in This is the transpose of the LPP projection matrix. To match the original input features Reconstructing feature vectors with consistent dimensions, dimensions .

[0079] 3.5 Design of the total loss function for fused KMM Loss

[0080] The core function of the total loss function is to coordinate and adjust various performance indicators of the model. It integrates the reconstruction loss, the KL divergence loss of SGPVI and the KMM loss. The reconstruction loss ensures the reconstruction accuracy of the model, the KL divergence loss of SGPVI constrains the rationality of the potential distribution, and the KMM loss solves the distribution offset problem between the training set and the test set. The three are optimized together to improve the robustness of the model in anomaly detection.

[0081] Reconstruction loss The mean squared error (MSE) is used for calculation based on the original input feature vector. With reconstructing feature vectors The formula is ,in For the first The first sample 3D eigenvalues The loss function measures the model's ability to reconstruct the input features, where the corresponding reconstructed feature values ​​are represented.

[0082] SGPVI's KL divergence loss Let KL divergence be the mean of the posterior and prior distributions of SGPVI. The calculation process is as follows: ,in The KL divergence calculated in Section 3.3 is a loss function that constrains the latent distribution to approximate the prior distribution, thus avoiding model overfitting.

[0083] The core function of KMM Loss is to align the LPP dimensionality-reduced feature distributions of the training and test sets, thus addressing the performance degradation caused by data distribution shifts in practical applications. Let the LPP dimensionality-reduced feature matrix of the training set be... , The number of samples in the training set is ; the LPP dimensionality reduction feature matrix of the test set is . , To determine the number of samples in the test set, KMM introduces sample weights. Adjust the contribution of each sample in the training set to the distribution alignment so that the kernel mean of the training set and the test set tends to be consistent in the regenerated kernel Hilbert space (RKHS).

[0084] The formula for calculating KMM Loss is as follows: ,in For RKHS, The kernel function for KMM (consistent with the RBF kernel of SGPVI, i.e.) (to ensure the consistency of kernel function parameters) Let be the weight vector of the training set samples, with the following constraints: and This ensures the non-negativity and physical meaning of the weights, meaning the sum of the weights equals the number of samples in the training set.

[0085] For ease of calculation, the KMM Loss is expanded into matrix form: ,in Kernel matrix between training set samples, dimension ; Training set - test set kernel matrix, dimensions , Kernel matrix between test set samples, dimension , It is a vector consisting entirely of 1s.

[0086] The total loss function is a weighted sum of the three types of losses, and the formula is as follows: The weights of the SGPVI KL divergence loss are... The weights of KMM Loss The weight combination was determined through multiple cross-validations and can achieve the optimal balance between reconstruction accuracy, the rationality of potential distribution, and the distribution alignment effect.

[0087] S4: Model Training and Optimization Integrating Multiple Algorithms

[0088] The core of the model training process is to minimize the total loss function through iterative optimization. Simultaneously, it achieves coordinated updates of LPP, IAF, SGPVI, and KMM Loss, ensuring parameter adaptability of each algorithm module, and ultimately outputting a convergent network anomaly detection model with strong generalization ability.

[0089] First, the standardized structured feature data output by S2 is divided into training, validation, and test sets in a 7:2:1 ratio. Stratified sampling is used during this division to ensure that the ratio of normal to abnormal samples in each dataset is consistent with the original data. For each of the divided datasets, the corresponding dimensionality-reduced feature matrices are generated using the LPP dimensionality reduction process described in Section 3.1. , , , serving as input features for each stage.

[0090] The iterative update process during training is executed per Epoch (training round), and the specific steps are as follows:

[0091] 1. Fix the parameters of IAF SGPVI induction point With kernel parameters Sample weights of KMM Based on the current neighborhood weight matrix The projection matrix of LPP is updated through the generalized eigenvalue solution process described in Section 3.1. This ensures that the dimensionality reduction features can adapt to the local structure capture requirements of the current model state;

[0092] 2. Fixed LPP projection matrix Sample weights of KMM posterior mean based on SGPVI output The IAF parameter adjustment formula in Section 3.2 can be used. Update IAF parameters Simultaneously, the encoder's weight matrix is ​​updated by combining the backpropagation gradient of the encoder. , , , With bias vector , , , ;

[0093] 3. Fixed LPP projection matrix Parameters of IAF Based on the current latent matrix By minimizing the KL divergence of SGPVI Update the SGPVI homing point With kernel parameters Simultaneously update the mean vector of the induced points. ;

[0094] 4. Fixed LPP projection matrix SGPVI parameters ( , , LPP dimensionality reduction feature matrix based on training and validation sets , By minimizing KMM Loss Optimize the sample weights of KMM Satisfying the constraints and ;

[0095] 5. Calculate the total loss function based on all current parameters. The Adam optimizer is used for backpropagation, synchronously updating all learnable parameters, including LPP. Encoder weights and biases, IAF SGPVI and KMM The weights and biases of the decoder. The initial learning rate of the Adam optimizer is set to 0.001, and the learning rate decays to 0.9 every 20 training epochs, adaptively adjusting the learning rate to balance convergence speed and stability;

[0096] 6. After each training round, the dimensionality-reduced feature matrix of the validation set is... Input the model and calculate the total loss on the validation set. ,like If the model does not decrease for five consecutive rounds, the early stopping strategy is triggered to stop training and prevent overfitting.

[0097] The convergence and validation of the model must meet multiple metrics: the reconstruction loss of the validation set. The reconstruction accuracy meets the standard; the KL divergence loss of SGPVI is... SGPVI convergence test; KMM Loss The distribution alignment is satisfactory. If any of the above indicators are not met, the weighting coefficients will be adjusted. (Step size 0.1) and (Step size 0.05), re-execute the training process until all convergence metrics meet the requirements, and output the trained model.

[0098] S5: Determine the anomaly detection threshold

[0099] This step, based on the model trained with S4 and the validation set data, calculates the final anomaly determination error by fusing the basic reconstruction error and the KMM distribution correction term. The optimal anomaly detection threshold is determined by combining multiple methods to ensure that the threshold can accurately distinguish between normal and abnormal network data and provide a reliable judgment standard.

[0100] First, error calculation is performed using the validation set data partitioned in S2: the standardized structured feature data of the validation set is preprocessed according to the S2 preprocessing procedure to generate LPP dimensionality reduction features, which are then input into the model trained in S4. The model then undergoes a series of processes including encoder, IAF transform, SGPVI posterior estimation, and decoder reconstruction to calculate the basic reconstruction error for each validation set sample. KMM distribution correction term Then follow the fusion formula ,in The final anomaly detection error for each sample is calculated. .

[0101] Threshold determination employs a two-tiered strategy: a basic threshold method and a progressive optimization approach. The basic threshold method involves statistically analyzing all samples in the validation set. Distribution, using the 95th percentile as the initial threshold The quantile selection is based on the statistical assumption that 95% of normal samples should fall within the threshold range, ensuring that most normal data will not be misclassified as abnormal. Progressive threshold optimization addresses the dynamic characteristics of data distribution by introducing an adaptive adjustment mechanism: first, normal samples in the validation set are manually labeled, and then these normal samples are calculated... mean with standard deviation Set the initial adaptive threshold Then, using a sliding window of 100 validation set samples as the interval, the normal samples within the window were recalculated. and and update the threshold This ensures that the threshold can adapt to local fluctuations in the data distribution and improves the robustness of the threshold.

[0102] The threshold validation phase requires verifying the effectiveness of the threshold through the anomaly sample recognition rate: manually annotated anomaly samples from the validation set are input into the model, and their... And count those exceeding the threshold The percentage of samples; if this percentage reaches 90% or higher, it indicates that the threshold can effectively identify abnormal samples, the threshold is deemed effective, and the final anomaly detection threshold is output. If the proportion is lower than 90%, adjust the quantile proportion of the basic threshold method by a step size of 5%, such as from 95% to 90% or 100%, or adjust the standard deviation multiple of the adaptive threshold method by a step size of 0.5%, such as from 3σ to 2.5σ or 3.5σ. Recalculate the threshold and verify it again until the abnormal sample identification rate requirement is met.

[0103] S6: Network Anomaly Detection and Inference Integrating Multiple Algorithms

[0104] The core of the inference process is to use the trained model parameters to perform anomaly detection on the real-time collected and preprocessed network data, which requires strict reuse of the LPP projection matrix optimized during the training phase. Encoder parameters, IAF parameters SGPVI parameters ( , , KMM sample weights To ensure consistency between the inference and training processes, a KMM distribution correction term is introduced to optimize anomaly detection errors and improve detection accuracy. The real-time detection and deployment process is as follows: Figure 4 As shown.

[0105] The specific calculation steps of the reasoning process are as follows:

[0106] 1. Real-time feature preprocessing and dimensionality reduction: For the raw network data collected in real time by S1, data cleaning, feature extraction, and Z-score standardization are performed according to the preprocessing workflow of S2 to generate standardized feature vectors. (dimension) ); based on the trained LPP projection matrix Calculate real-time dimensionality reduction features (dimension) This step reuses the dimensionality reduction logic from the training phase to ensure the consistency of feature representation;

[0107] 2. Latent Space Transformation and Posterior Estimation: [The text abruptly ends here, likely due The trained encoder is input, and the hidden feature vector is calculated through two hidden layers. This leads to the generation of initial potential distribution parameters. and Initial latent vectors are generated through reparameterization techniques. ( );Will Input training completed The final latent vector is generated using the transformation formula in Section 3.2 for the layer IAF. ;Will Input SGPVI, based on the induction points after training. nuclear parameters Calculate the posterior mean With posterior covariance ;

[0108] 3. Reconstructed Feature Generation and Error Calculation: Posterior Covariance Based on SGPVI Generate decoder input vector ( );Will The trained decoder is input and processed through two hidden layers to generate reconstructed dimensionality-reduced features. Reconstructing features by restoring them to their original dimensions. ; Calculate the basic reconstruction error This error measures the difference between real-time data and reconstructed data;

[0109] 4. KMM Distribution Correction Term Calculation: To address the distribution offset issue between real-time data and training data, a KMM distribution correction term is introduced. The calculation process is as follows ,in For the training set LPP dimensionality reduction features of each sample For the training set KMM weights for each sample, The RBF kernel function is determined for the training phase; The larger the value, the greater the difference in distribution between real-time data and training data, which needs to be corrected in the anomaly detection process;

[0110] 5. Final Anomaly Detection Error and Anomaly Classification: The final anomaly detection error is obtained by fusing the basic reconstruction error with the KMM distribution correction term. ,in As the weight of the correction term; The anomaly detection threshold determined by S5 If a comparison is made, If the real-time sample is determined to be normal network data; This was determined to be abnormal network data.

[0111] 6. Anomaly Classification and Result Output: Based on The size of the outlier data is used to classify the outlier data, and the classification threshold is based on different types of outlier samples in the validation set. Distribution determined: Mild anomalies correspond to Minor anomalies include occasional port scan attempts and short-term traffic fluctuations; moderate anomalies correspond to... General anomalies include multiple failed login attempts and continuous low-traffic attacks; severe anomalies correspond to... It detects serious anomalies such as DDoS attacks and large-scale data breaches; it outputs the detection results for each real-time sample, including sample identifier (timestamp + source IP + destination IP) and basic reconstruction error. KMM distribution correction term Final anomaly determination error The judgment results include: normal, mildly abnormal, moderately abnormal, and severely abnormal.

[0112] S7: Validation of outliers and optimization of model feedback

[0113] To achieve continuous improvement in model performance, this step involves manual verification and closed-loop optimization based on the detection results of S6. The actual detection of true and false anomalies is fed back into the model training process to update model parameters and anomaly detection thresholds, forming a continuous improvement mechanism of detection-verification-optimization.

[0114] The manual verification process for anomalies needs to establish a standardized review procedure: Anomaly detection results output by S6 should be categorized by severity and pushed to the network operations and maintenance platform, prioritizing severe and moderate anomaly samples. Operations and maintenance personnel can view detailed information about the samples through the platform, including timestamps, source IPs, destination IPs, and event types. , , The system, combined with background information such as network topology and security policies, undergoes manual review to label samples with their true characteristics: if a sample exhibits genuine abnormal behavior such as network attacks or unauthorized operations, it is labeled as "True Anomaly" and the specific anomaly type is recorded, such as DDoS attack, port scanning, login cracking, data leakage, or privilege abuse; if a sample exhibits normal network behavior but is mistakenly identified as abnormal, it is labeled as "False Anomaly" and the sample's characteristic information is recorded. Possible causes of composition and misjudgment include sudden surges in normal traffic and data from new business scenarios.

[0115] In the model optimization data preparation stage, a high-quality optimized training set needs to be constructed: add manually labeled true anomaly samples and false anomaly samples to the original training set, and regenerate standardized structured feature data according to the preprocessing process of S2, including data cleaning, feature extraction, and Z-score standardization. At the same time, generate the corresponding dimensionality reduction feature matrix according to the LPP dimensionality reduction process of S3. Then, redivide the optimized training set, optimized validation set, and optimized test set in a ratio of 7:2:1 to ensure that the sample distribution of the new dataset is consistent with the actual application scenario.

[0116] During the model fine-tuning phase, parameters for all core algorithm modules are updated: the model trained with S4 is fine-tuned using an optimized training set. The fine-tuning parameters are set to a learning rate of 0.0001, which is 1 / 10 of the initial training learning rate, to avoid large parameter fluctuations. The training runs for 20 epochs with a batch size of 32, and the Adam optimizer is still used. During the fine-tuning process, not only are the weights and biases of the encoder and decoder updated, but the projection matrix of LPP is also updated simultaneously. Parameters of IAF SGPVI induction point With kernel parameters Sample weights of KMM This ensures that the parameters of all fusion algorithms are adapted to the feature distribution of the new samples.

[0117] The threshold update phase recalculates the optimal threshold based on the fine-tuned model: the optimized validation set data is input into the fine-tuned model, and the threshold for each sample is calculated according to the S5 process. Using the same threshold determination method, either the basic threshold method or the progressive optimization method, the anomaly detection threshold is recalculated. Replace the original threshold .

[0118] To ensure the timeliness and effectiveness of optimization, a fixed optimization frequency is set: model fine-tuning and threshold updates are automatically triggered every 1,000 manually labeled samples or every 7 days. If a major cybersecurity incident occurs during this period, such as a large-scale DDoS attack or a new type of vulnerability exploitation attack, emergency optimization can be manually triggered to ensure that the model can quickly adapt to new attack types and network environment changes.

[0119] S8: Deployment of Real-time Network Anomaly Detection

[0120] This step deploys the optimized S7 model and thresholds to the real-time detection system, building a full-process real-time detection system that includes real-time acquisition, rapid processing, accurate judgment, timely alarm, and data feedback, meeting the real-time security monitoring needs of the target network.

[0121] The real-time data processing stage must ensure low latency and high reliability: Network traffic and log data are captured in real-time using the acquisition method described in S1, with real-time data reception and caching achieved through a streaming processing framework (such as Flink); the real-time preprocessing module cleans, extracts features, and standardizes the raw data according to the process in S2. The processing employs a parallel computing architecture to ensure that the preprocessing latency for a single data point is ≤1 second, meeting the time requirements for real-time detection; the preprocessed standardized feature data is directly input into the LPP dimensionality reduction module, reusing the projection matrix optimized in S7. Generate dimensionality reduction features This provides input for real-time inference.

[0122] The real-time inference detection process strictly follows the S6 inference workflow: Input the optimized encoder to generate the initial latent vector. ;through Layer IAF transformation obtained Input the SGPVI module to calculate the posterior mean. With covariance ; Generate decoder input based on posterior covariance The decoder reconstructs the result. ;calculate , and and with the updated threshold Real-time comparisons are performed to determine anomalies. The entire inference process is accelerated by GPU computing, ensuring an inference latency of ≤500 milliseconds for a single data entry and supporting a data processing capacity of over 1000 data entries per second.

[0123] The anomaly alert push process needs to implement tiered alerts and multi-channel notifications: when abnormal data is detected, the anomaly information, including sample identifier, anomaly type, etc., should be sent in real time. The system pushes alerts to the network operations and maintenance platform, including the time of occurrence, the IP addresses and ports involved, etc. The platform displays different colored alarm icons according to the severity of the anomaly (mild, moderate, severe). At the same time, it sends alarm notifications to operations and maintenance personnel through multiple channels such as SMS, email, and WeChat Work robots. Severe anomalies trigger telephone voice alarms to ensure that operations and maintenance personnel can respond in a timely manner.

[0124] The data feedback process provides data support for subsequent model optimization: all sample data detected in real time, including normal and abnormal samples, are recorded in the offline data pool in the S1 storage format. Every 24 hours, the data in the offline data pool is synchronized to the model optimization data warehouse to accumulate manually labeled samples and trigger model fine-tuning, forming a closed-loop mechanism of real-time detection-data feedback-model optimization-real-time detection, ensuring the continuous improvement of the detection system's performance.

[0125] In some embodiments, to address the problem that traditional LPP fixed neighborhood construction methods cannot adapt to the dynamic distribution of network data, a dynamic neighborhood radius + attention-weighted LPP neighborhood construction mechanism is introduced. Specifically, the fixed k-nearest neighbor or fixed Gaussian kernel bandwidth setting is abandoned, and the neighborhood radius is adaptively adjusted through the local density of samples: for the i-th sample... Calculate its local density , The initial bandwidth is set to 0.3; if If the density is greater than the mean of the entire sample, then the neighborhood radius is reduced to... Enhance the resolution of local structures; if If it is less than the mean, then increase the radius to... This avoids missing neighborhoods for sparse samples. Simultaneously, attention weights are introduced. The corrected neighborhood weight matrix is ​​calculated as follows: The final neighborhood weights are updated to This mechanism dynamically adjusts the neighborhood range through density awareness and combines attention weights to highlight the contribution of key neighborhood samples, thereby improving the adaptability of LPP dimensionality reduction to the sparsity and dynamism of network data. Its core improvement lies in upgrading the static neighborhood to a dynamic adaptive neighborhood, solving the dimensionality reduction distortion problem of traditional LPP in scenarios of network burst traffic and sparse attack samples.

[0126] In some embodiments, based on the improvements to the dynamic neighborhood LPP described above, the collaborative mechanism between IAF and SGPVI is further optimized, and a hierarchical adversarial IAF (HIAF) structure is proposed. Based on the original four-layer IAF, it is divided into three sub-layers according to the latent space dimension: a low-dimensional feature layer (1-20 dimensions), a mid-dimensional feature layer (21-40 dimensions), and a high-dimensional feature layer (41-64 dimensions). Each sub-layer is followed by a lightweight discriminator. , The discriminator's input is the output of the corresponding sublayer IAF. Subspace posterior mean of SGPVI , Within the sub-layer dimension, the goal is to distinguish between true latent features (output during the stable training phase) and generated latent features (output of the current iteration); the goal of IAF is to generate latent features that can fool the discriminator, thus forming adversarial training. The adversarial loss function is... The total loss function adds an adversarial loss term. , This improvement enhances the expressive power of latent features in each dimension through hierarchical adversarial mechanisms, while making the output of IAF more closely match the posterior distribution of SGPVI, further strengthening the synergistic adaptability of the two and solving the problem of insufficient modeling of high-dimensional latent space features by traditional IAF.

[0127] In some embodiments, based on the improvement of hierarchical adversarial IAF, the kernel function cooperation mechanism of KMM and SGPVI is further optimized, and an adaptive kernel parameter sharing and dynamic weight adjustment strategy is proposed. In traditional schemes, the kernel parameters of KMM and SGPVI are consistent but fixed; this embodiment increases the kernel bandwidth... Set as a learnable parameter and dynamically adjust it using the posterior covariance of SGPVI: ,in The posterior covariance of the sublayer corresponding to SGPVI. To achieve adaptive sharing of kernel parameters at the sub-layer level, dynamic weight coefficients are introduced. Adjust the proportion of KMM Loss in the total loss. , For the initial KMM Loss, When the distribution alignment is good, When the distribution shift is small, the weights are reduced to avoid over-constraint; when the distribution shift is large, the weights are increased to ensure alignment. This improvement achieves dual dynamic coordination of kernel parameters and loss weights of KMM and SGPVI, solving the problem that traditional fixed weights cannot adapt to dynamic changes in data distribution, and further improving the model's robustness to distribution shifts.

[0128] In some embodiments, based on the S3-based dynamic neighborhood LPP and hierarchical adversarial IAF, a phased heterogeneous optimization strategy is proposed to address the parameter convergence asynchrony problem during multi-algorithm fusion training. The training process is divided into three stages: the first stage (rounds 1-30) is the feature adaptation stage, where the adversarial module of IAF and the induction point of SGPVI are fixed, and only the projection matrix of LPP is optimized. Encoder basic parameters and KMM weights The first stage (rounds 31-80) uses the Ranger optimizer with a learning rate of 0.002, focusing on adapting the LPP dimensionality reduction features to the encoder input. The second stage (rounds 31-80) is the co-modeling stage, enabling IAF adversarial training and SGPVI induced point updates, fixing the LPP parameters, switching the optimizer to AdamW with a learning rate of 0.001, and introducing gradient accumulation (accumulation steps 4) to improve the stability of adversarial training. The third stage (after round 81) is the global convergence stage, simultaneously optimizing all parameters, with the learning rate decaying according to a cosine annealing strategy, a minimum learning rate of 0.0001, and introducing a dynamic threshold for early stopping. If the validation set... The decline has been less than 5 times for 5 consecutive rounds. ,and , If the condition is met, early stopping is triggered. This strategy addresses the problems of local optima traps and convergence asynchrony in multi-algorithm fusion training by focusing on the optimization objectives of different modules in stages and combining heterogeneous optimizers with dynamic early stopping conditions, thereby improving the stability and convergence quality of model training.

[0129] In some embodiments, based on the aforementioned phased heterogeneous optimization strategy, a multi-scale gradient clipping and noise injection enhancement mechanism is further introduced to improve the model's generalization ability. Differential gradient clipping thresholds are adopted for the gradient characteristics of different modules: the gradient clipping threshold for the LPP module is set to 1.0, for the encoder / decoder to 0.5, for the IAF adversarial module to 0.3, and for SGPVI to 0.8, to avoid gradient vanishing or exploding caused by a single threshold. Simultaneously, during the backpropagation process in each training phase, dynamic Gaussian noise is injected into the gradient of the LPP projection matrix and the kernel parameter gradient of the SGPVI. ,in , The noise intensity is highest in the first stage of the total training rounds and gradually decreases as training progresses. This mechanism balances the gradients of each module through multi-scale gradient pruning and dynamic noise injection to break the local optima of parameter convergence, further improving the model's generalization ability and solving the problem of overfitting in multi-algorithm fusion models.

[0130] In some embodiments, a threshold optimization method based on a phased optimization model is proposed to address the sensitivity of traditional quantile thresholds to abnormal sample distributions. First, for normal samples in the validation set... Fitting Gamma distribution Shape parameters Scale parameters Solve using maximum likelihood estimation: , ,in Normal sample mean Standard deviation; for outlier samples Fitting the Gumbel distribution Position parameters Scale parameters The solution is obtained through maximum likelihood estimation. Then, the optimal threshold is sought. This reduces the false positive rate for normal samples. ,Right now The initial threshold is obtained by solving the integral equation through numerical integration. Simultaneously, the false negative rate for outlier samples is calculated. If the false negative rate is greater than 0.1, the false positive rate is adjusted to 0.03 and the solution is recalculated to finally obtain the optimal threshold that balances the false positive and false negative rates. This method accurately characterizes the error distribution of normal and abnormal samples through bidistribution fitting, which is more statistically reasonable than the traditional quantile method and solves the problem of inaccurate threshold determination when the sample distribution is uneven.

[0131] In some embodiments, based on the aforementioned dual-distribution fitting threshold, a time-dependent threshold adaptive adjustment mechanism is further introduced to adapt to the temporal characteristics of the network data. A lightweight LSTM model is trained using the temporal data of the validation set, with the first 5 time steps as input. With the corresponding threshold The output is the threshold adjustment coefficient at the current time. Adjust the formula to ,in To limit the adjustment range and avoid excessive threshold fluctuations, the training objective of LSTM is to minimize the weighted sum of the false positive and false negative rates at the current threshold. During the inference phase, the previous 5 time steps are input in real time. Based on historical thresholds, output via LSTM Adjust the current threshold This mechanism captures the temporal evolution of errors through time-series modeling, enabling the threshold to adapt to temporal fluctuations in network data, such as error changes caused by traffic fluctuations during peak periods, thereby further improving the temporal adaptability of anomaly detection.

[0132] In some embodiments, a lightweight inference acceleration and intermediate result caching mechanism is proposed based on the time-adaptive threshold of S5 to address the latency issue of real-time inference. For the sparse inference process of SGPVI, an induced point pre-clustering + incremental update strategy is adopted: during the training phase, induced points are pre-clustered... Clustered into 5 clusters using K-means, during inference, only the kernel function value between the current sample and its cluster center is calculated, instead of all inducing points, reducing the kernel matrix computation by 70%; simultaneously, the LPP dimensionality reduction features of the most recent 100 samples are cached. With IAF transformation results If the Euclidean distance between the new sample and the cached sample is less than the threshold If a sample is identified as similar, the cached sample can be reused directly. Inputting SGPVI skips the encoder and IAF calculation process. Furthermore, pruning optimization is applied to the decoder, removing weights with absolute values ​​less than [a certain value]. The number of model parameters was reduced by 45% through the improved connectivity. Through these optimizations, the single-sample inference latency was reduced from 500 milliseconds to 180 milliseconds, meeting the requirements for high-concurrency real-time detection.

[0133] In some embodiments, based on the aforementioned lightweight acceleration mechanism, a cross-sample inference consistency verification mechanism is further introduced to improve the accuracy of anomaly detection. Samples for real-time inference are grouped by time windows (10 seconds), and the consistency of each sample within the window is calculated. With window mean deviation Simultaneously, calculate the feature similarity between the current sample and the first three samples within the window. , If the sample is judged as abnormal, but and If the result is consistent with the majority of samples within the window, it should be corrected to a suspected anomaly and requires further verification with subsequent window samples; if it is judged to be normal, but... and If the difference from surrounding samples is significant, the sample is classified as a suspected anomaly. For suspected anomaly samples, the complete inference process is initiated and the posterior entropy of SGPVI is calculated. ,like If the posterior uncertainty is high, the result is pushed to manual review; otherwise, the corrected result is output. This mechanism reduces the single-sample misjudgment rate and solves the problem of inference accuracy loss that may be caused by lightweight acceleration by using consistency checks within the time window and posterior entropy uncertainty assessment.

[0134] The embodiments described above are for illustrative purposes only and are not intended to limit the invention. Therefore, any changes in numerical values ​​or substitutions of equivalent elements should still fall within the scope of this invention.

[0135] The above detailed description will enable those skilled in the art to understand that the present invention can indeed achieve the aforementioned objectives and has complied with the provisions of the Patent Law.

[0136] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention. The above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

[0137] It should be noted that the above description of the process is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the process under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0138] The basic concepts have been described above. Obviously, for those skilled in the art who have read this application, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore, such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this application.

[0139] Furthermore, this application uses specific terms to describe its embodiments. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different positions in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0140] Furthermore, those skilled in the art will understand that aspects of this application can be described and illustrated through several patentable types or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Therefore, aspects of this application can be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. All of the above hardware or software can be referred to as a “unit,” “module,” or “system.” Furthermore, aspects of this application can take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.

[0141] The computer program code required for the operation of each part of this application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C, VB.NET, and Python; general programming languages ​​such as C; Visual Basic, Fortran2103, Perl, COBOL2102, PHP, and ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. This program code can run entirely on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as Software as a Service (SaaS).

[0142] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this application are not intended to limit the order of the processes and methods of this application. Although some currently considered useful embodiments of the invention have been discussed in the foregoing disclosure by way of various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the substance and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a purely software solution, such as an installation on an existing server or mobile device.

[0143] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this approach of the present application should not be construed as reflecting an intention that the claimed subject matter requires more features than expressly recited in each claim. Rather, the subject of the invention should possess fewer features than in any single embodiment described above.

Claims

1. A network anomaly detection method based on variational autoencoders, characterized in that, Includes the following steps: S1: Collect network traffic data and network log data in the target network environment to form raw network data; S2: Preprocess the raw network data and output standardized structured feature data; S3: Construct a variational autoencoder model integrating multiple algorithms. Dimensionality reduction is achieved by performing dimensionality reduction on standardized structured feature data through local preserving projection, and this reduced feature is input into the encoder to generate an initial latent vector. This vector is then subjected to an invertible transformation via an inverse autoregressive flow to obtain the transformed latent vector. A sparsified posterior estimate is performed on the transformed latent vector using variational inference through a sparse Gaussian process. The decoder reconstructs the features based on the sparsified posterior estimate. The reconstruction loss is calculated from the standardized structured feature data and the reconstructed features. The KL divergence loss is calculated from the prior and posterior distributions based on the variational inference of the sparse Gaussian process. The kernel mean matching loss is calculated based on the dimensionality reduction features. The reconstruction loss, KL divergence loss and kernel mean matching loss are fused to construct the total loss function. S4: Train a variational autoencoder model that integrates multiple algorithms using training set data from standardized structured feature data. Iteratively optimize and minimize the total loss function, update the relevant parameters of local preserving projection, inverse autoregressive flow, sparse Gaussian process variational inference and kernel mean matching loss until the model converges and output the trained model. S5: Input the validation set data from the standardized structured feature data into the trained model, calculate the anomaly detection error, and determine the anomaly detection threshold based on the distribution characteristics of the anomaly detection error; S6: Preprocess the real-time collected network data to obtain real-time standardized structured feature data, input it into the trained model, calculate the real-time final anomaly judgment error, compare it with the anomaly detection threshold, and output the anomaly detection result.

2. The network anomaly detection method based on variational autoencoder according to claim 1, characterized in that, Also includes: S7: Manually verify the anomaly detection results and label them with the true labels. Add the labeled samples to the original training data, repeat steps S2, S4 and S5, update the model parameters and anomaly detection threshold, and obtain the optimized model and the updated anomaly detection threshold. S8: Deploy the optimized model and updated anomaly detection threshold to the real-time detection system to achieve real-time anomaly monitoring of the target network.

3. The network anomaly detection method based on variational autoencoder according to claim 1, characterized in that, In S3, there is a two-way interaction between local preservation projection and sparse Gaussian process variational inference. Specifically, sparse Gaussian process variational inference updates the neighborhood weight matrix of local preservation projection through its inter-sample kernel matrix. Local preservation projection then recalculates the projection matrix based on the updated neighborhood weight matrix, thereby correcting the dimensionality reduction features.

4. The network anomaly detection method based on variational autoencoder according to claim 1, characterized in that, In S3, there is a parameter feedback adjustment process between the inverse autoregressive flow and the sparse Gaussian process variational inference. Specifically, the initial value of the transformation parameter of the inverse autoregressive flow is randomly initialized, and during the training process, it is adjusted by the posterior mean of the output of the sparse Gaussian process variational inference, so that the transformation of the inverse autoregressive flow fits the potential distribution characteristics modeled by the sparse Gaussian process variational inference.

5. The network anomaly detection method based on variational autoencoder according to claim 1, characterized in that, The kernel mean matching loss in S3 is calculated based on the dimensionality reduction features of the locally preserved projection output. By introducing training set sample weights, the kernel mean of the dimensionality reduction features of the training set and the test set tends to be consistent in the regenerated kernel Hilbert space, thus achieving distribution alignment.

6. The network anomaly detection method based on variational autoencoder according to claim 1, characterized in that, The iterative optimization process for training the variational autoencoder model that integrates multiple algorithms in S4 is as follows: in each round of training, the projection matrix of the local preserving projection, the transformation parameters of the inverse autoregressive flow, the induction point and kernel parameters of the sparse Gaussian process variational inference, and the sample weights of the kernel mean matching loss are updated sequentially, and then all learnable parameters are updated synchronously based on the total loss function.

7. The network anomaly detection method based on variational autoencoder according to claim 1, characterized in that, The convergence criteria for the model in S4 are: the reconstruction loss is less than the preset reconstruction loss threshold, the KL divergence loss is less than the preset KL divergence loss threshold, the kernel mean matching loss is less than the preset kernel mean matching loss threshold, and the total loss has not decreased for a preset number of consecutive rounds.

8. The network anomaly detection method based on variational autoencoder according to claim 1, characterized in that, In S5, the determination of the anomaly detection threshold adopts a progressive optimization strategy. Specifically, the initial threshold is first determined based on the quantiles of the final anomaly judgment error of all samples in the validation set. Then, the mean and standard deviation of the final anomaly judgment error of normal samples in the validation set are calculated through a sliding window, and the anomaly detection threshold is dynamically updated.

9. The network anomaly detection method based on variational autoencoder according to claim 1, characterized in that, In S6, the real-time final anomaly determination error is a weighted sum of the basic reconstruction error and the kernel mean matching distribution correction term. The basic reconstruction error is calculated based on the real-time standardized structured feature data and the corresponding reconstruction features, while the kernel mean matching distribution correction term is calculated based on the distribution difference between the real-time dimensionality reduction features and the training set dimensionality reduction features.

10. The network anomaly detection method based on variational autoencoder according to claim 1, characterized in that, The anomaly detection results in S6 include four levels: normal, mild anomaly, moderate anomaly, and severe anomaly. The classification is based on the extent to which the real-time final anomaly judgment error exceeds the anomaly detection threshold.

Images