Abnormal traffic detection method based on sparse matrix

Abstract

The invention relates to an abnormal traffic detection method based on a sparse matrix, and belongs to the technical field of computer network security. Through sparse matrix compression storage, dynamic feature updating and an efficient calculation framework, the resource consumption is reduced, and the detection precision is improved. Experiments show that the detection delay is less than 30ms under the 100Gbps flow environment, the storage occupation is reduced by 87.5%, the F1-score on CIC-IDS2017 and UNSW-NB15 data sets reaches more than 96%, and the false alarm rate is less than 2%. The method is suitable for large-scale network scenes such as a 5G core network, a cloud data center and the Internet of Things, and the real-time anomaly detection efficiency and the resource utilization rate can be remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer network security, and particularly relates to an abnormal traffic detection method based on a sparse matrix. Background Art

[0002] With the rapid development of network technology, various new types of malicious attacks have emerged continuously. How to improve the classification accuracy of malicious network traffic is the key to improving the performance of abnormal traffic detection and defending against malicious attacks. The existing abnormal traffic detection methods have the following problems. (1) Efficiency bottleneck: Traditional methods (such as models based on PCA and LSTM) need to process the complete traffic matrix, and the computational complexity is O(n 3 ), which cannot meet the real-time requirements. (2) Storage redundancy: More than 90% of the original traffic data are redundant features (such as zero values and repeated protocol fields), and full-scale storage occupies too much resources. (3) Poor dynamic adaptability: Static models are difficult to adapt to the dynamic changes of network traffic distribution, resulting in an increase in the false positive rate (FPR). (4) Noise sensitivity: Tiny fluctuations in sparse data are easily misjudged as abnormal (such as the confusion between DDoS attacks and burst traffic). Summary of the Invention

[0003] (I) Technical Problems to be Solved

[0004] The technical problems to be solved by the present invention are: aiming at the problems of low detection efficiency, storage redundancy, and poor dynamic adaptability in high-dimensional sparse network traffic data, designing an abnormal traffic detection method to reduce resource consumption and improve detection accuracy.

[0005] (II) Technical Solutions

[0006] To solve the above technical problems, the present invention provides an abnormal traffic detection method based on a sparse matrix, including the following steps:

[0007] Step 1: Selection and preprocessing of traffic data:

[0008] (1) Dataset selection: Select the public datasets CIC-IDS2017 dataset and UNSW-NB15 dataset as the benchmark datasets;

[0009] (2) Preprocess the data in the selected datasets: First is feature cleaning: Read the data in the CSV format file, extract the feature columns, including five-tuples, traffic size, and timestamp, and delete all-zero columns, constant columns, and highly correlated features; then perform standardization: Perform Z-score standardization on continuous features and one-hot encoding on discrete features;

[0010] Step 2: Construction and dynamic update of the sparse matrix:

[0011] (1) Sparse matrix construction: Directly map the cleaned and standardized data into a sparse matrix to achieve sparsification, only retaining the coordinates of non-zero elements, and storing it in the Compressed Sparse Row (CSR) format;

[0012] When constructing the sparse matrix, the input features are the features of the preprocessed network traffic metadata, including: five-tuple, traffic size, and time series features; in the matrix mapping rule used for constructing the sparse matrix, the matrix rows are traffic batches within a time slice, and each traffic batch contains multiple network flows; the matrix columns are feature dimensions; the non-zero filling value is the value after feature standardization;

[0013] (2) Dynamic update of the sparse matrix: After constructing the sparse matrix, periodically perform dynamic updates on the sparse matrix. The dynamic update mechanism includes a sliding window mechanism and an incremental insertion mechanism; the sliding window mechanism retains data within a time window T, and expired data is automatically eliminated through the LRU policy; the data within the time window is stored in a circular buffer; the incremental insertion mechanism is to locate the insertion position of the new data block through a hash index table when a new data block needs to be inserted;

[0014] Step Three: Feature dimensionality reduction and abnormal traffic detection:

[0015] (1) Perform feature dimensionality reduction on the sparse matrix M obtained in Step Two: Project the sparse matrix M into a low-dimensional space to obtain a projection matrix W. The elements in the projection matrix follow a {-1, 0, +1} distribution, and the elements of the projection matrix are taken as +1 or -1 with probability and taken as 0 with probability where s ≥ 1, which is used to control the sparsity;

[0016] (2) Adopt an abnormal scoring strategy to detect abnormal traffic for the data represented by the projection matrix obtained after feature dimensionality reduction: Use the reconstruction error method to calculate the reconstruction error Score(x) for the original traffic data x of each traffic batch to characterize the difference between x and the projection matrix W':

[0017] Score(x) = ||x - W'W' T x||2

[0018] where x represents the original traffic data of a single traffic batch, that is, a certain row in the sparse matrix M. If the reconstruction error Score(x) exceeds the dynamic threshold, then x is determined as abnormal traffic.

[0019] Preferably, the constant column is a fixed protocol field, the highly correlated feature is a feature with a Pearson coefficient > 0.9, and the discrete feature is such as the protocol type.

[0020] Preferably, in step two (1), the data after sparsification processing is further divided: according to the time sequence, the data is divided into a training set, a validation set, and a test set, with proportions of 70%, 20%, and 10% in sequence. The training set is used to train the dimensionality reduction model and the anomaly detection model. The validation set is used to optimize the sliding window size and the regularization coefficient. The test set is used for the performance evaluation of anomaly traffic detection.

[0021] Preferably, the five-tuple includes the source IP, destination IP, source port, destination port, and protocol. The traffic size includes the total number of bytes and the number of packets. The time series features include the flow start / end timestamps and the packet interval statistical values.

[0022] Preferably, in step two (2), a batch merge and write strategy is also adopted to accumulate the data of multiple time slices and then batch-update the sparse matrix.

[0023] Preferably, in step three (1), an L1 regularization constraint is also introduced to the sparse PCA to generate an objective function to optimize the projection matrix:

[0024]

[0025] The iterative update formula during optimization is as follows:

[0026] Z ij =(sign(W ij +U ij ))max(|W ij +U ij |-λ / ρ,0)

[0027] U←U+(W - Z)

[0028] The alternating direction method of multipliers ADMM is used to alternately update W, Z, and the dual variable U. After iterating a certain number of times, it converges. Fix Z and update W to solve the least squares problem under the orthogonal constraint. Fix W and update Z, and apply soft thresholding shrinkage to impose sparsity. Finally, the optimized projection matrix W’ is obtained, where Z ij is an element in matrix Z, W ij is an element in matrix W, U ij is an element in matrix U, ρ is the penalty parameter, λ is the L1 regularization coefficient, and λ is used to control the intensity of the sparsity of the projection matrix.

[0029] Preferably, when optimizing the projection matrix in step three (1), it converges after 100 iterations; in step three (2), the isolation forest model or the one-class SVM model is also used to perform a secondary analysis on the reconstruction error Score(x); when using the isolation forest model, 100 trees are constructed, with a maximum depth of 10 for each tree, and the anomaly score is determined by the path length; when using the one-class SVM model, the RBF kernel function is selected.

[0030] Preferably, the dynamic threshold is adjusted using the Exponentially Weighted Moving Average (EWMA) formula, and the EWMA formula is as follows:

[0031] θ t = αθ t-1 + (1 - α)·score t

[0032] The initial dynamic threshold θ0 is set to the 95th percentile of the sample scores in the training set. In the training set, 95% of the sample scores are lower than θ0, and 5% of the sample scores are higher than θ0; where θ t represents the dynamic threshold at the current moment, t represents the current time step, i.e., the current moment, t - 1 represents the previous time step; α is the smoothing factor; score t is the reconstruction error at the current moment.

[0033] Preferably, after constructing and dynamically updating the sparse matrix in step two, M is first divided into multiple sub - blocks, and then the feature dimensionality reduction and abnormal traffic detection in step three are performed on each sub - block, with each GPU thread processing one sub - block.

[0034] The present invention also provides a system for implementing the above - mentioned method.

[0035] (III) Beneficial Effects

[0036] Aiming at the problems of low detection efficiency, storage redundancy, and poor dynamic adaptability in high - dimensional sparse network traffic data, the present invention provides a lightweight, highly real - time and dynamically adaptable abnormal traffic detection method. Through sparse matrix compression storage, dynamic feature update, and an efficient computing framework, resource consumption is reduced and detection accuracy is improved. The present invention is realized through the following technical solutions: First, the original traffic data is mapped into a sparse matrix representation, and only non - zero elements and indexes are stored. Combined with the sliding window mechanism, the matrix is dynamically updated, reducing the storage overhead to 10% - 30% of the traditional method; Second, sparse random projection or improved Sparse Principal Component Analysis (Sparse PCA) is used for feature dimensionality reduction, compressing the dimension to 5% - 10% of the original data; Further, the abnormal score is calculated based on the reconstruction error or a lightweight model (such as Isolation Forest), and a dynamic threshold adjustment strategy is designed to reduce false alarms through Exponentially Weighted Moving Average (EWMA). Experiments show that the detection latency of the present invention is less than 30 ms in a 100 Gbps traffic environment, the storage occupancy is reduced by 87.5%, the F1 - score reaches over 96% on the CIC - IDS2017 and UNSW - NB15 datasets, and the false alarm rate is less than 2%. The present invention is applicable to large - scale network scenarios such as 5G core networks, cloud data centers, and the Internet of Things, and can significantly improve the real - time abnormal detection efficiency and resource utilization rate. Description of the Drawings

[0037] Figure 1 This is the overall flowchart of the method of the present invention. Specific embodiments

[0038] To make the objectives, content and advantages of the present invention clearer, the following further describes in detail the specific embodiments of the present invention with reference to the accompanying drawings and embodiments.

[0039] Aiming at problems such as low detection efficiency, storage redundancy and poor dynamic adaptability in high-dimensional sparse network traffic data, the present invention provides a lightweight, highly real-time and dynamically adaptable abnormal traffic detection method. Through sparse matrix compression storage, dynamic feature update and an efficient computing framework, resource consumption is reduced and detection accuracy is improved. The present invention is realized through the following technical solutions: First, the original traffic data is mapped into a sparse matrix representation, and only non-zero elements and indexes are stored. Combining a sliding window mechanism to dynamically update the matrix, reducing the storage overhead to 10% - 30% of the traditional method; Second, sparse random projection or improved sparse principal component analysis (Sparse PCA) is used for feature dimensionality reduction, compressing the dimension to 5% - 10% of the original data; Further, an anomaly score is calculated based on the reconstruction error or a lightweight model (such as Isolation Forest), and a dynamic threshold adjustment strategy is designed to reduce false alarms through exponential weighted moving average (EWMA).

[0040] Reference Figure 1 , a method for detecting abnormal traffic based on a sparse matrix according to the present invention specifically includes the following steps:

[0041] Step 1. Selection and preprocessing of traffic data:

[0042] (1) Dataset selection and feature analysis: The method of the present invention selects the publicly available datasets CIC-IDS2017 dataset and UNSW-NB15 dataset as benchmark datasets, which complement each other and cover diverse attack scenarios. The CIC-IDS2017 dataset was developed in cooperation between the Communications Security Establishment Canada (CSE) and the Canadian Institute for Cybersecurity (CIC), and contains normal traffic and various attacks (DDoS, Brute Force, etc.), with rich features and accurate timestamps. The UNSW-NB15 dataset was developed by the Centre for Cyber Security at the University of New South Wales Canberra, Australia, and is characterized by a mixture of normal and novel attack traffic, providing network flow-level statistical features and being suitable for sparse matrix modeling.

[0043] (2) Preprocess the data in the selected dataset: First is feature cleaning: Read the data in the CSV file, extract the feature columns (such as five-tuples, traffic size, timestamp), delete all-zero columns, constant columns (such as fixed protocol fields), and highly correlated features (Pearson coefficient > 0.9); then perform standardization: Perform Z-score standardization on continuous features and one-hot encoding on discrete features (such as protocol type).

[0044] Step 2. Construction and dynamic update of the sparse matrix:

[0045] (1) Sparse matrix construction: Directly map the cleaned and standardized data into a sparse matrix to achieve sparsification, only retaining the non-zero element coordinates (i, j, value), and the storage format adopts the CSR (Compressed Sparse Row) format, with a compression efficiency improvement of 3 - 5 times; finally is data partitioning: Partition the data into a 70% training set, 20% validation set, and 10% test set in chronological order. The training set is used to train the dimensionality reduction model and the anomaly detection model, the validation set is used to tune hyperparameters (such as sliding window size, regularization coefficient), and the test set is used for final performance evaluation (verify the effect of the present invention after Step 3) to ensure no time leakage.

[0046] When constructing the sparse matrix, the input features are the features of the preprocessed network traffic metadata, including: five-tuples (source IP, destination IP, source port, destination port, protocol), traffic size (total number of bytes, number of packets), and temporal features (flow start / end timestamp, packet interval statistics). In the matrix mapping rule used when constructing the sparse matrix, the matrix rows are traffic batches within a time slice (such as 1 minute), and each traffic batch contains multiple network flows; the matrix columns are feature dimensions (such as 65 dimensions are retained after processing by CIC-IDS2017); the non-zero filling value is the value after feature standardization (normalization) (such as the number of packets is standardized to [0, 1]).

[0047] (2) Dynamic update of the sparse matrix: After constructing the sparse matrix, periodically perform dynamic updates on the sparse matrix. The dynamic update mechanism includes a sliding window mechanism and an incremental insertion mechanism. The principle of the sliding window mechanism is to retain data within a time window T (configurable from 1 minute to 1 hour), and expired data is automatically eliminated through the LRU (Least Recently Used) strategy. The data within the time window is stored in a circular buffer to ensure O(1) time complexity for insertion and deletion. The optimization method of the incremental insertion mechanism is that when a new data block needs to be inserted, the new data block quickly locates the insertion position through a hash index table, avoiding full matrix scanning. Adopt a batch merge and write strategy, accumulate data from multiple time slices and then batch update the matrix to reduce disk I / O overhead.

[0048] The storage efficiency comparison between the sparse matrix constructed in this step and the traditional dense matrix is shown in the following table:

[0049] Table 1 Storage Efficiency Comparison

[0050]

[0051]

[0052] Step 3: Feature Dimensionality Reduction and Abnormal Traffic Detection:

[0053] (1) Perform feature dimensionality reduction on the sparse matrix M obtained in Step 2: First is sparse random projection. Project the sparse matrix onto a low-dimensional space to obtain a projection matrix W ∈ The elements of the projection matrix follow a {-1, 0, +1} distribution, and the elements of the projection matrix are taken as +1 or -1 with probability and taken as 0 with probability where s ≥ 1 is used to control the sparsity level (usually s = 1), and the computational complexity of the projection matrix is reduced to O(nk). Then, improve sparse PCA (Principal Component Analysis) by introducing L1 regularization constraints to obtain an objective function (i.e., the mathematical expression of improved sparse PCA) for optimizing the projection matrix:

[0054]

[0055] Iterative update formula:

[0056] Z ij =(sign(W ij +U ij ))max(|W ij +U ij |-λ / ρ, 0)

[0057] U ← U+(W - Z)

[0058] Use the Alternating Direction Method of Multipliers (ADMM) to alternately update W, Z, and the dual variable U. After 100 iterations, it converges. Fix Z and update W to solve the least squares problem under orthogonal constraints. Fix W and update Z, and apply soft-thresholding shrinkage to impose sparsity. Finally, obtain the optimized projection matrix W'. Among them, Z ij is the element in matrix Z, W ij is the element in matrix W, U ij is the element in matrix U, ρ is the penalty parameter, λ is the L1 regularization coefficient, and λ is used to control the strength of matrix sparsity. The larger λ is, the sparser W is (more elements are zero), and the value of λ is adjusted according to the validation set.

[0059] (2) Anomaly scoring strategy is adopted to detect abnormal traffic for the data represented by the sparse matrix after feature dimensionality reduction: The present invention uses the reconstruction error method to calculate the reconstruction error Score(x) for the original traffic data x of each traffic batch (i.e., the difference between the original traffic data x and the optimized projection matrix W', and normal traffic can be better reconstructed by the dimensionality reduction model with a smaller error):

[0060] Score(x) = ||x - W'W' T x||2

[0061] Among them, x represents the original traffic data of a single traffic batch, that is, a certain row in the sparse matrix M. If the reconstruction error Score(x) exceeds the dynamic threshold, it is determined as abnormal.

[0062] In the present invention, a lightweight model such as Isolation Forest or One-Class SVM (OC-SVM) is also used to perform secondary analysis on the reconstruction error Score(x) to improve the detection robustness. 100 trees are constructed in the Isolation Forest, with a maximum depth of 10 for each tree, and the anomaly score is determined by the path length. The kernel function of the One-Class SVM selects RBF, and the parameter υ = 0.01.

[0063] The dynamic threshold is adjusted using the Exponentially Weighted Moving Average (EWMA) formula as follows:

[0064] θ t = αθ t-1 +(1 - α)·score t

[0065] The initial threshold θ0 is set to the 95th percentile of the sample scores in the training set. In the training set, 95% of the samples have scores lower than θ0, and 5% of the samples have scores higher than θ0. θ t represents the dynamic threshold at the current moment, t represents the current time step (i.e., the current moment), t - 1 represents the previous time step, and there is a 5-minute interval between them. The dynamic threshold is updated every 5 minutes to adapt to the traffic distribution drift; α is the smoothing factor. In this embodiment, α = 0.95; score t is the reconstruction error at the current moment;

[0066] Optimization and acceleration implementation: The present invention uses the CUDA (Compute Unified Device Architecture) parallel computing framework and the cuSPARSE library to accelerate sparse matrix multiplication. That is, after constructing and dynamically updating the sparse matrix in step two, M is first divided into 128×128 sub-blocks, and then the feature dimensionality reduction and abnormal traffic detection in step three are performed on each sub-block, and each GPU thread processes one sub-block. Shared memory is used to cache frequently accessed data to reduce the global memory access latency.

[0067] Embodiment

[0068] The core implementation process of the present invention revolves around three major technical modules: sparse matrix modeling, dynamic feature update, and efficient dimensionality reduction detection, and realizes real-time anomaly detection of high-dimensional sparse network traffic through the following four steps.

[0069] Step 1: Data preprocessing and sparsification: First, clean and standardize the original traffic data, and convert it into a sparse matrix representation, only retaining non-zero elements and their coordinates, compressing the storage space to 10% - 30% of the traditional method. The input is the original traffic data (CSV files of CIC-IDS2017 / UNSW-NB15), and the output is the standardized sparse matrix. Then, perform dynamic update on the sparse matrix. The schematic code is shown in Table 2 below;

[0070] Step 2: Dynamic update of the sparse matrix: The core logic of the sliding window is to maintain a time window of a fixed size, and automatically eliminate the oldest data when new data enters. The data structure is implemented using a circular buffer (CircularBuffer) or a double-ended queue (Deque), and the time complexity is O(1). The incremental insertion optimization is accelerated through a hash index. A hash table is established for each column to record the positions of non-zero elements, and the insertion is directly located. Use batch writing to accumulate data of multiple time slices and then batch update the matrix to reduce I / O overhead.

[0071] Table 2 Pseudo-code for data preprocessing and dynamic update of the sparse matrix

[0072]

[0073] Step 3: Feature dimensionality reduction and anomaly detection model: Improve the sparse PCA algorithm, and the objective function is:

[0074]

[0075] Randomly generate an orthogonal matrix W. Then, use alternating optimization of W and Z (the objective function contains two variables W and Z, and coordinate the consistency of W and Z). Specifically, fix W by applying sparsity through soft thresholding, update Z, and use singular value decomposition (SVD) to ensure orthogonality to fix Z and update W.

[0076] The anomaly score and the dynamic threshold are realized through the reconstruction error calculation and the dynamic threshold formula. The formula is shown in Step 3, and the code example is as follows:

[0077] Table 3 Pseudo-code for feature dimensionality reduction and anomaly detection model

[0078]

[0079]

[0080] GPU-accelerated sparse matrix multiplication. The code example is as follows:

[0081] Table 4 Pseudocode for GPU-accelerated sparse matrix multiplication

[0082]

[0083] The key parameter configurations are shown in the following table:

[0084] Table 5 Key parameter configurations

[0085]

[0086] Compared with the prior art, the method of the present invention has the following advantages: In terms of storage efficiency, in the CIC-IDS2017 dataset test, the storage occupancy of the sparse matrix is only 12.5% of that of the traditional method. In terms of real-time performance, when a single node processes 100 Gbps of traffic, the detection latency ≤ 30 ms (220 ms for the traditional PCA method). In terms of detection accuracy, on the UNSW-NB15 dataset, the F1-score reaches 96.2%, and the false positive rate (FPR) is reduced to 1.8%. The present invention is applicable to large-scale network scenarios such as 5G core networks, cloud data centers, and the Internet of Things, and can significantly improve the real-time anomaly detection efficiency and resource utilization rate.

[0087] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and deformations can be made, and these improvements and deformations should also be regarded as the protection scope of the present invention.

Claims

1. An abnormal traffic detection method based on a sparse matrix, characterized in that, It includes the following steps: Step 1. Selection and preprocessing of traffic data: (1) Dataset selection: Select the publicly available datasets CIC-IDS2017 dataset and UNSW-NB15 dataset as the benchmark datasets; (2) Preprocess the data in the selected datasets: First is feature cleaning: Read the data in the CSV format file, extract the feature columns, including five-tuples, traffic size, and timestamp, and delete the all-zero columns, constant columns, and highly correlated features; then perform standardization: Perform Z-score standardization on continuous features and one-hot encoding on discrete features; Step 2. Construction and dynamic update of the sparse matrix: (1) Sparse matrix construction: Directly map the cleaned and standardized data into a sparse matrix to achieve sparsification, only retain the non-zero element coordinates, and the storage format adopts the compressed sparse row CSR format; When constructing the sparse matrix, the input features are the features of the preprocessed network traffic metadata, including: five-tuples, traffic size, and temporal features; in the matrix mapping rule used when constructing the sparse matrix, the matrix rows are the traffic batches within a time slice, and each traffic batch contains multiple network flows; the matrix columns are the feature dimensions; the non-zero filling value is the value after feature standardization; (2) Dynamic update of the sparse matrix: After constructing the sparse matrix, periodically perform dynamic updates on the sparse matrix. The dynamic update mechanism includes a sliding window mechanism and an incremental insertion mechanism; the sliding window mechanism retains data within the time window T, and the expired data is automatically eliminated through the LRU strategy; the data within the time window is stored in a circular buffer; the incremental insertion mechanism is to locate the insertion position of the new data block through the hash index table when a new data block needs to be inserted; Step 3. Feature dimensionality reduction and abnormal traffic detection: (1)Perform feature dimensionality reduction on the sparse matrix M obtained in step two: project the sparse matrix M into a low-dimensional space to obtain a projection matrix W. The elements in the projection matrix follow a distribution of {-1, 0, +1}, and the elements of the projection matrix take +1 or -1 with probability and take 0 with probability , where s ≥ 1 is used to control the sparsity level; (2) Adopt an abnormal scoring strategy to detect abnormal traffic for the data represented by the projection matrix obtained after feature dimensionality reduction: Use the reconstruction error method to calculate the reconstruction error Score(x) for the original traffic data x of each traffic batch to characterize the difference between x and the projection matrix W'; Score(x)=||x - W'W' T x||2 where x represents the original traffic data of a single traffic batch, that is, a certain row in the sparse matrix M. If the reconstruction error Score(x) exceeds the dynamic threshold, then x is determined to be abnormal traffic.

2. The method according to claim 1, characterized in that, The constant column is a fixed protocol field, the highly correlated feature is a feature with a Pearson coefficient > 0.9, and the discrete feature is such as the protocol type.

3. The method according to claim 1, wherein In (1) of Step 2, the sparsified data is also divided: The data is divided into a training set, a validation set, and a test set in chronological order, with proportions of 70%, 20%, and 10% respectively. The training set is used to train the dimensionality reduction model and the abnormal detection model, the validation set is used to optimize the sliding window size and regularization coefficient, and the test set is used to evaluate the performance of abnormal traffic detection.

4. The method according to claim 1, wherein The five-tuples include source IP, destination IP, source port, destination port, and protocol. The traffic size includes the total number of bytes and the number of packets. The temporal features include the flow start / end timestamp and the packet interval statistical value.

5. The method according to claim 1, wherein In (2) of Step 2, a batch merge and write strategy is also adopted to accumulate data from multiple time slices and then batch update the sparse matrix.

6. The method according to claim 1, wherein In (1) of Step 3, the L1 regularization constraint is also introduced to the sparse PCA to generate an objective function to optimize the projection matrix: The iterative update formula during optimization is as follows: Z ij = (sign(W ij + U ij )) max(|W ij + U ij | - λ / ρ, 0) U←U+(W-Z) The Alternating Direction Method of Multipliers (ADMM) is used to alternately update W, Z, and the dual variable U. After iterating a certain number of times, it converges. Fix Z and update W to solve the least squares problem under the orthogonal constraint. Fix W and update Z, and apply soft thresholding shrinkage to impose sparsity. Finally, the optimized projection matrix W’ is obtained, where Z ij is an element in matrix Z, and W ij is an element in matrix W, and U ij is an element in matrix U. ρ is the penalty parameter, and λ is the L1 regularization coefficient. λ is used to control the intensity of the sparsity of the projection matrix.

7. The method according to claim 1, wherein In (1) of Step 3, the optimization of the projection matrix converges after 100 iterations; in (2) of Step 3, the Isolation Forest model or the One-Class SVM model is also used for secondary analysis of the reconstruction error Score(x); when using the Isolation Forest model, 100 trees are constructed, with the maximum depth of each tree being 10, and the anomaly score is determined by the path length; when using the One-Class SVM model, the RBF kernel function is selected.

8. The method according to claim 1, wherein The dynamic threshold is adjusted using the Exponentially Weighted Moving Average (EWMA) formula, and the EWMA formula is as follows: θ t = αθ t-1 +(1 - α)·score t The initial dynamic threshold θ0 is set to the 95th percentile of the sample scores in the training set. In the training set, 95% of the samples have scores lower than θ0, and 5% of the samples have scores higher than θ0. Among them, θ t represents the dynamic threshold at the current moment, t represents the current time step, i.e., the current moment, and t−1 represents the previous time step; α is the smoothing factor; score t is the reconstruction error at the current moment.

9. The method according to claim 1, characterized in that, After constructing and dynamically updating the sparse matrix in Step 2, M is first divided into multiple sub-blocks, and then the feature dimension reduction and abnormal traffic detection in Step 3 are performed on each sub-block, with each GPU thread processing one sub-block.

10. A system for implementing the method according to any one of claims 1 to 9.

Images