ISP network abnormal state automatic detection method based on deep learning algorithm

Abstract

The invention provides an ISP network abnormal state automatic detection method based on a deep learning algorithm, and the method comprises the steps: collecting multi-dimensional data of an ISP network, extracting time sequence features and topological structure features from the multi-dimensional data, splicing the time sequence features and topological structure features into a fusion feature vector, inputting the fusion feature vector into an ISP network state abnormality monitoring model based on an LSTM-GNN-VAE hybrid model, and carrying out the automatic detection of the abnormal state of the ISP network in an encoder. The LSTM sub-network is used for processing the time sequence characteristic data to obtain a time sequence coding vector, the GNN sub-network is used for processing the topological structure characteristics to obtain a topological coding vector, the time sequence coding vector and the topological coding vector are spliced to obtain a fusion vector, the fusion vector is mapped to a potential space, and a mean value and a variance of potential distribution are output; the decoder generates potential vectors through a re-parameterization technique according to the mean value and variance of the potential distribution, maps the potential vectors back to an original feature space, and reconstructs a comprehensive feature vector. Through targeted processing of the multi-dimensional data, the model is optimized in combination with a plurality of set optimization strategies, and the accuracy of ISP network state anomaly detection is effectively improved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of network state anomaly detection, in particular to an ISP network anomaly state automatic detection method based on a deep learning algorithm. BACKGROUND

[0002] ISP network, also known as Internet Service Provider network, refers to a network infrastructure system constructed and operated by an Internet service provider, and the core function thereof is to provide users with services for connecting to the Internet. The ISP network helps users access the global Internet through physical facilities (such as optical cables, routers, etc.), and may also provide basic services such as information browsing and file transmission. However, in the prior art, there are significant technical shortcomings in the anomaly detection of the ISP network state, which is difficult to adapt to the complex environment of the modern network. For example, the traditional detection method relies on manually defined rules and thresholds, which is difficult to cope with the dynamic changes of the network environment, so that the anomaly detection result is inaccurate.

[0003] The core bottleneck is the insufficient data processing capacity. Among the PB-level traffic generated by the ISP network daily, the proportion of encrypted traffic has exceeded 75%, and the traditional deep packet inspection technology is invalid, which can only rely on limited metadata features, making it difficult to identify hidden threats such as APT attacks. At the same time, the heterogeneity of edge nodes and backbone networks (such as the difference between Internet of Things devices and data center links) makes it impossible for a single detection model to adapt to all scenarios, and the average positioning time of link faults exceeds 40 minutes. These defects directly cause serious consequences. The bandwidth loss caused by the missed report of DDoS attacks exceeds 10 million yuan per year, the cost of manual troubleshooting caused by false positives accounts for 20% of the operation and maintenance expenses, and the user complaint rate increases by 15% due to network instability. Therefore, improving the accuracy of ISP network state anomaly detection is not only a demand for technological upgrading, but also an inevitable choice for ISPs to reduce operating costs and ensure service quality. SUMMARY

[0004] To solve the above problems, the application provides an ISP network anomaly state automatic detection method based on a deep learning algorithm, which analyzes the time sequence features and topological structure features of the ISP network, and judges whether the ISP network state is abnormal according to the calculated reconstruction error. The judgment method is simple and accurate.

[0005] To achieve the above purpose, the technical solution adopted by the application is: The application provides an ISP network anomaly state automatic detection method based on a deep learning algorithm, which includes the following steps: S1, collect multi-dimensional data of the ISP network and pre-process the multi-dimensional data, wherein the multi-dimensional data includes network traffic data, device state data and topological structure data; S2, feature engineering is performed on the pretreated data, time sequence features and topological structure features are extracted, and the fusion feature vector is formed by splicing the time sequence features and the topological structure features; S3, a hybrid model based on LSTM-GNN-VAE is constructed as an ISP network state abnormality monitoring model, the fusion feature vector is taken as input, the latent distribution of the data is learned through an encoding and decoding process, and the input data is reconstructed, and whether the ISP network state is abnormal is judged according to the calculated reconstruction error.

[0006] Further, the collected data is preprocessed, and the preprocessing includes data cleaning, data conversion, data standardization and data alignment.

[0007] The step S3 includes the following steps: S31, an encoder containing an LSTM subnetwork, a GNN subnetwork and a fusion encoding layer is constructed, the time sequence encoding vector is obtained after the time sequence features are processed by the LSTM subnetwork, the topological encoding vector is obtained after the topological structure features are processed by the GNN subnetwork, the fusion encoding layer splices the time sequence encoding vector and the topological encoding vector to obtain a fusion vector, the fusion vector is mapped to a latent space, and the mean and variance of the latent distribution are outputted; S32, a decoder with the same structure as the encoder is constructed, the latent vector is generated by a reparameterization technique according to the mean and variance of the latent distribution, the comprehensive feature vector is reconstructed by mapping back to the original feature space.

[0008] Preferably, the loss function of the model adopts a weighted sum of reconstruction loss and KL divergence loss, and the formula is: Loss=α×Loss Reconstruction +β×Loss KL ; In the formula, α and β are weight coefficients, Loss Reconstruction represents the reconstruction loss, the mean square error (MSE) is used to calculate the difference between the reconstructed features and the original features; Loss KL represents the KL divergence loss, which is used to measure the difference between the latent distribution and the standard normal distribution, so as to make the latent distribution close to the standard normal distribution.

[0009] Further, the method further includes: S4, when it is judged that the ISP network state is abnormal, the abnormal mode is classified according to the feature performance and the distribution of the reconstruction error when the abnormality occurs, different processing strategies are formulated for different abnormal modes, and the abnormal modes include flow abnormality, device abnormality and topological abnormality.

[0010] Further, the method further includes: S5, the hyperparameters of the mixed model are optimized through a Bayesian optimization algorithm, the F1 value of the model on the verification set is taken as the optimization target, an Gaussian process proxy model is constructed, and the optimal combination of hyperparameters is iteratively searched, the blindness of hyperparameter tuning is reduced, and the model performance is improved.

[0011] Further, in step S5, a dynamic threshold adjustment strategy is provided, which can dynamically adjust the abnormality judgment threshold based on the current network state, and the threshold adjustment formula is: T=T0×(1+k×L); In the formula, T0 is the initial threshold, k is the adjustment coefficient, and L is the network load index (0≤L≤1).

[0012] Further, in step S5, an incremental learning mechanism is introduced, so that the model can continuously learn new network state data and adapt to the changes of the network.

[0013] Further, in step S5, an online updating mechanism is established, and the model is evaluated regularly, and when the detection performance of the model decreases to a certain threshold, the model updating process is triggered.

[0014] The beneficial effects of the present application are: The present application provides an ISP network abnormal state automatic detection method based on a deep learning algorithm, multi-dimensional data of an ISP network is collected, time sequence features and topological structure features are extracted therefrom, and are spliced into a fusion feature vector, which is input into an ISP network state abnormality monitoring model based on a mixed model of LSTM-GNN-VAE, in the encoder, the LSTM subnetwork is used to process the time sequence feature data to obtain a time sequence encoding vector, the GNN subnetwork is used to process the topological structure features to obtain a topological encoding vector, the time sequence encoding vector and the topological encoding vector are spliced to obtain a fusion vector which is mapped to a latent space, and the mean and variance of the latent distribution are output; the decoder generates a latent vector through a reparameterization technique according to the mean and variance of the latent distribution, maps back to the original feature space, and reconstructs a comprehensive feature vector. Through targeted processing of multi-dimensional data, the model is optimized by setting various optimization strategies, and the accuracy of ISP network state abnormality detection is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The present application provides an ISP network abnormal state automatic detection method based on a deep learning algorithm. DETAILED DESCRIPTION

[0016] Please refer to Figure 1 The present application provides an ISP network abnormal state automatic detection method based on a deep learning algorithm, which includes the following steps: S1, collect multi-dimensional data of ISP network and preprocess it, including network traffic data, device status data and topology data; With the help of traffic collection devices deployed in network core nodes and edge nodes, such as high-performance network probes, network traffic data is collected in real time using packet capture technology (such as libpcap). The collection content covers source IP address, destination IP address, source port, destination port, protocol type (TCP, UDP, ICMP, etc.), packet length, transmission rate, session duration, packet arrival time interval, etc. The collection frequency is dynamically adjusted according to the network bandwidth. In high-bandwidth core links, the collection frequency is set to 10ms / time; in edge links, the collection frequency is set to 50ms / time, while supporting targeted collection of traffic for specific protocols and ports.

[0017] Through SNMP (Simple Network Management Protocol) v3 version, the state information of devices such as routers, switches and servers in the network is collected. The collected device status data includes CPU usage, memory utilization, disk I / O rate, port traffic, port error rate (such as CRC error, frame error), device temperature, power state, etc. The collection period is 30 seconds / time, and for key core devices, the collection period is shortened to 10 seconds / time, ensuring that the device running state can be grasped in time.

[0018] Comprehensively use LLDP (Link Layer Discovery Protocol), CDP (Cisco Discovery Protocol) and network topology discovery tools to obtain network topology data in real time. The collection content includes the connection relationship between devices, link type (such as Ethernet, optical fiber), link bandwidth, link delay, device location information, device model and manufacturer information, etc. The network topology is scanned automatically every 5 minutes, and when a device is detected to be connected, removed or the link state changes, the collection process is triggered immediately to update the topology database.

[0019] The collected data is preprocessed, including data cleaning, data conversion, data standardization and data alignment.

[0020] Through data cleaning, remove duplicate, error and incomplete data records to ensure the accuracy and integrity of the data. For example, using rule-based and statistical methods, remove duplicate packet records in traffic data, abnormal values (such as CPU usage greater than 100%) in device status data, and invalid connection information in topology data.

[0021] Convert non-numeric features (such as protocol type) in network traffic data into numeric features using one-hot encoding, for example, encode TCP protocol as [1, 0, 0], UDP protocol as [0, 1, 0], and ICMP protocol as [0, 0, 1].

[0022] The numerical type data (such as data packet length, transmission rate, CPU usage, etc.) is standardized by using the min-max standardization method to map the data to the interval [0, 1] to eliminate the influence of dimension and facilitate model learning.

[0023] The network traffic data, device status data and topology data are aligned in time based on the timestamp to form a unified time series data set. For data with different collection frequencies, interpolation or downsampling methods are used for processing to ensure that all data are consistent in the time dimension. At the same time, the data correlation is established to associate the device status data with the corresponding network traffic data and topology data to form a complete network state description.

[0024] S2, feature engineering is performed on the preprocessed data to extract time series features and topology features, and the fusion feature vector is formed by splicing them; When extracting time series features, the preprocessed network traffic data and device status data are divided into time windows, and statistical features such as mean, variance, maximum, minimum, peak, valley, slope (reflecting data trend), autocorrelation coefficient (reflecting data periodicity) are calculated in each time window. For example, for the transmission rate of network traffic, the average transmission rate, rate variance and maximum transmission rate in each window are calculated. An LSTM network is constructed to extract deep features from time series data. The sequence data in the time window is input to the LSTM, and the LSTM network contains 3 hidden layers with 256, 128 and 64 neurons respectively, and uses ReLU activation function. Through the gating mechanism of the LSTM network, the long-term dependence and dynamic change law of the network state data are captured, and finally a 64-dimensional deep time series feature vector is output.

[0025] Based on the graph structure of the topology data, the degree of the node (the number of edges connected to the node), the betweenness centrality (the importance of the node in the shortest path of the network), the closeness centrality (the inverse of the average distance from the node to other nodes), the clustering coefficient (the degree of connection between the neighbors of the node), and other features such as the bandwidth utilization of the link and the link delay fluctuation are calculated. A GCN (Graph Convolutional Network) model in GNN (Graph Neural Network) is used to extract deep topology features. The GCN model contains 2 convolution layers, the first layer takes the node features and adjacency matrix as input and outputs a 128-dimensional feature vector; the second layer further processes the output of the first layer and outputs a 64-dimensional deep topology feature vector. Through the message passing mechanism of the GCN model, the information of the neighbor nodes is aggregated to learn the structural role and associated features of the node in the entire network topology.

[0026] The 64-dimensional time sequence features extracted by the LSTM and the 64-dimensional topological structure features extracted by the GCN are fused in a feature splicing manner to form a 128-dimensional comprehensive feature vector as the input of the subsequent hybrid model.

[0027] S3, a hybrid model based on LSTM-GNN-VAE is constructed as an ISP network state anomaly monitoring model, taking the fused feature vector as the input, learning the latent distribution of the data through the encoding and decoding process, and reconstructing the input data, and judging whether the ISP network state is abnormal according to the calculated reconstruction error; The step S3 includes the following steps: S31, an encoder containing an LSTM subnetwork, a GNN subnetwork and a fusion encoding layer is constructed, the time sequence encoding vector is obtained after the time sequence feature is processed by the LSTM subnetwork, the topological encoding vector is obtained after the topological structure feature is processed by the GNN subnetwork, the fusion encoding layer splices the time sequence encoding vector and the topological encoding vector to obtain a fusion vector, and the fusion vector is mapped to a latent space to output the mean and variance of the latent distribution; Specifically, the LSTM subnetwork receives the time sequence feature, processes it through two fully connected layers (the number of neurons is 128 and 64 respectively), and obtains the time sequence encoding vector; the GNN subnetwork receives the topological structure feature, processes it through two fully connected layers (the number of neurons is 128 and 64 respectively), and obtains the topological encoding vector. The fusion encoding layer splices the time sequence encoding vector and the topological encoding vector to obtain a 128-dimensional fusion vector, and then maps the fusion vector to a latent space through two fully connected layers (the number of neurons is 96 and 64 respectively) to output the mean and variance of the latent distribution. S32, a decoder with the same structure as the encoder is constructed, the latent vector is generated through the reparameterization trick according to the mean and variance of the latent distribution, mapped back to the original feature space, and the comprehensive feature vector is reconstructed.

[0028] Specifically, the decoding part generates the latent vector through the reparameterization trick according to the mean and variance of the latent distribution, and then maps the latent vector back to the original feature space through three fully connected layers (the number of neurons is 64, 96 and 128 respectively) to reconstruct a 128-dimensional comprehensive feature vector. The loss function of the model adopts the weighted sum of the reconstruction loss and the KL divergence loss, and the formula is: Loss=α×Loss Reconstruction +β×Loss KL ; In the formula, α and β are weight coefficients, Loss Reconstruction represents the reconstruction loss, the mean square error (MSE) is used to calculate the difference between the reconstructed feature and the original feature; Loss KLrepresents the KL divergence loss, which is used to measure the difference between the latent distribution and the standard normal distribution, and encourages the latent distribution to approach the standard normal distribution.

[0029] In the model training phase, a large amount of normal network state data is used for training, so that the model learns the latent distribution of the normal data. In the inference phase, the extracted comprehensive features are input into the model, and the reconstruction error (i.e. reconstruction loss) is calculated. When the reconstruction error exceeds the set threshold, it is judged that the network state is abnormal.

[0030] S4, when judging that the ISP network state is abnormal, according to the feature performance and the distribution of the reconstruction error when the anomaly occurs, the abnormal mode is classified, and corresponding processing strategies are formulated for different abnormal modes, the abnormal mode includes traffic anomaly, device anomaly and topology anomaly.

[0031] The abnormal mode is divided into: Traffic anomaly: such as DDoS attack (manifested as a sudden increase in traffic of a specific IP address, time series feature reconstruction error significantly increases), port scanning (manifested as a large number of connection requests of different ports in a short time, session feature anomaly), abnormal traffic burst (such as transmission of data packets with abnormal large traffic) and the like. Device anomaly: such as device overload (CPU usage, memory utilization continuously too high, device state feature reconstruction error), device failure (port error rate rises sharply, device temperature is too high and the like). Topology anomaly: such as link interruption (corresponding link feature disappears, topology feature reconstruction error is large), network congestion (link bandwidth utilization rate approaches 100%, topology correlation feature is abnormal) and the like.

[0032] For different abnormal modes, corresponding processing strategies are formulated: Traffic anomaly processing: for DDoS attack, start traffic cleaning mechanism, filter attack traffic, limit access of attack source IP; for port scanning, ban source IP of abnormal scanning, strengthen monitoring of related ports; for abnormal traffic burst, adjust network bandwidth allocation, prioritize transmission of traffic of critical business.

[0033] Device anomaly processing: for device overload, migrate part of business to other devices to reduce device load; for device failure, issue alarm information to notify operation and maintenance personnel to repair or replace the device. Topology anomaly processing: for link interruption, enable standby link, re-plan routing path; for network congestion, adjust traffic routing, balance network load, and relieve congestion condition.

[0034] S5, the hyperparameters of the mixed model are optimized by a Bayesian optimization algorithm, the F1 value of the model on the validation set is taken as the optimization target, the optimal combination of hyperparameters is searched through the construction of a Gaussian process surrogate model, the blindness of hyperparameter tuning is reduced, and the model performance is improved.

[0035] The Bayesian optimization algorithm is used to optimize the hyperparameters of the LSTM-GNN-VAE hybrid model. The hyperparameters to be optimized include the number of hidden layer neurons of LSTM, the number of layers and the number of neurons in each layer of GNN, the latent space dimension of VAE, learning rate, batch_size, etc.

[0036] In step S5, a dynamic threshold adjustment strategy is provided, which can dynamically adjust the abnormality judgment threshold based on the current network state. The threshold adjustment formula is: T=T0×(1+k×L); In the formula, T0 is the initial threshold, k is the adjustment coefficient, and L is the network load index (0≤L≤1).

[0037] First, after the model training is completed, the initial threshold is determined according to the reconstruction error distribution of normal data (such as taking the 95% quantile of normal data reconstruction error). Then, the network load (such as average traffic, device average utilization) is monitored in real time, when the network load is high, the threshold is appropriately increased to reduce false positives; when the network load is low, the threshold is appropriately reduced to improve the sensitivity of anomaly detection.

[0038] In step S5, an incremental learning mechanism is introduced, which enables the model to continuously learn new network state data and adapt to network changes.

[0039] When new network data (including normal and abnormal data) is generated, the model is updated using incremental learning. Most of the model's parameters are retained, and only part of the model's layers (such as the top fully connected layer) are fine-tuned using new data for training, avoiding the model's forgetting of old knowledge, while improving the model's ability to adapt to new network states.

[0040] In step S5, an online updating mechanism is established, and the model is evaluated periodically. When the detection performance of the model decreases to a certain threshold, the model updating process is triggered.

[0041] Using the latest collected and preprocessed network data, combined with the incremental learning mechanism, the model is updated, and the updated model is deployed to the production environment, ensuring that the model always maintains good detection performance and adapts to the dynamic changes of the network.

[0042] The above embodiments are only used to describe the preferred embodiments of the present application, and are not used to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements made by those skilled in the art to the technical solutions of the present application shall fall within the protection scope of the claims of the present application.

Claims

1. An automatic detection method of an abnormal state of an ISP network based on a deep learning algorithm, characterized by, The method comprises the following steps: S1, collecting multi-dimensional data of an ISP network and preprocessing the same, wherein the multi-dimensional data comprises network traffic data, device status data and topology structure data; S2, performing feature engineering on the preprocessed data, extracting time sequence features and topology structure features, and splicing the same to form a fusion feature vector; S3, constructing a hybrid model based on LSTM-GNN-VAE as an ISP network state anomaly monitoring model, taking the fusion feature vector as input, learning the latent distribution of the data through encoding and decoding processes, and reconstructing the input data, and judging whether the ISP network state is abnormal according to the calculated reconstruction error. 2.The deep learning algorithm based ISP network anomaly state automatic detection method of claim 1, wherein, The collected data is preprocessed, and the preprocessing includes data cleaning, data conversion, data standardization and data alignment. 3.The deep learning algorithm based ISP network anomaly state automatic detection method of claim 1, wherein, Wherein, Step S3 comprises the following steps: S31, constructing an encoder comprising an LSTM subnetwork, a GNN subnetwork and a fusion encoding layer, obtaining a time sequence encoding vector through the LSTM subnetwork after processing the time sequence features, obtaining a topology encoding vector through the GNN subnetwork after processing the topology structure features, splicing the time sequence encoding vector and the topology encoding vector in the fusion encoding layer to obtain a fusion vector, mapping the fusion vector to a latent space, and outputting the mean and variance of the latent distribution; S32, constructing a decoder with the same structure as the encoder, generating a latent vector through a reparameterization technique according to the mean and variance of the latent distribution, mapping back to the original feature space, and reconstructing the comprehensive feature vector.

4. The method of claim 3, wherein the deep learning algorithm-based ISP network anomaly state automatic detection method is characterized by, The loss function of the model adopts a weighted sum of reconstruction loss and KL divergence loss, and the formula is: Loss = a x Loss Reconstruction + b x Loss KL ; wherein α and β are weight coefficients, Loss Reconstruction denotes the reconstruction loss, which calculates the difference between the reconstructed feature and the original feature using mean square error (MSE); Loss KL denotes the KL divergence loss, which is used to measure the difference between the latent distribution and the standard normal distribution, and promotes the latent distribution to be close to the standard normal distribution.

5. The method of claim 1, wherein the method further comprises: The method further comprises: S4, when judging that the ISP network state is abnormal, classifying the abnormal mode according to the feature performance and the distribution of the reconstruction error when the abnormality occurs, formulating a corresponding processing strategy for different abnormal modes, and the abnormal mode comprises traffic anomaly, device anomaly and topology anomaly.

6. The method of claim 1, wherein the deep learning algorithm-based ISP network anomaly state automatic detection method is characterized by, The method further comprises: S5, optimizing the hyperparameters of the hybrid model through a Bayesian optimization algorithm, taking the F1 value of the model on the validation set as the optimization target, constantly iterating to search for the optimal hyperparameter combination through the construction of a Gaussian process proxy model, reducing the blindness of hyperparameter tuning, and improving the performance of the model.

7. The method of claim 6, wherein the deep learning algorithm-based ISP network anomaly state automatic detection method is characterized by, In step S5, a dynamic threshold adjustment strategy is provided, which can dynamically adjust the abnormality judgment threshold based on the current network state, and the threshold adjustment formula is: T=T0×(1+k×L); In the formula, T0 is the initial threshold, k is the adjustment coefficient, and L is the network load index (0≤L≤1). 8.The deep learning algorithm based ISP network anomaly state automatic detection method of claim 6, wherein, In step S5, an incremental learning mechanism is introduced, so that the model can continuously learn new network state data and adapt to changes in the network. 9.The deep learning algorithm based ISP network anomaly state automatic detection method of claim 6, wherein, In step S5, an online updating mechanism is established, and the model is evaluated regularly. When the detection performance of the model decreases to a certain threshold, the model updating process is triggered.

Images