A method and system for OD flow estimation based on spatiotemporal network representation

Abstract

The present invention relates to the field of computer network communication technology, and in particular to an OD flow estimation method and system based on spatiotemporal network characterization, the steps of which are as follows: constructing a flow data set; constructing a flow self-characterization spatiotemporal network based on a spatiotemporal variational autoencoder and training it, introducing CNN and LSTM into the encoding part of the spatiotemporal variational autoencoder to express the spatiotemporal correlation characteristics in the network flow, using the decoding part of the spatiotemporal variational autoencoder as a generator, inputting link load data into the generator of the flow self-characterization spatiotemporal network, and solving the OD flow matrix of the target network using the link load data in the representation space of the generator. The present invention adopts a data-driven approach to solve, can make full use of the spatiotemporal characteristics in the network flow matrix, and can characterize its own space by learning the essential characteristics of the network flow and prior knowledge, so as to achieve efficient and high-precision estimation of OD flow and abnormal flow.

Description

Technical Field

[0001] The present invention relates to the field of computer network communication technology, and in particular to an OD flow estimation method and system based on spatiotemporal network representation. Background Art

[0002] As the scale of the network continues to expand, the various technologies and parameters used in the network have become more and more complex, and it has developed into a heterogeneous and open complex network, which has become increasingly difficult to effectively control and manage. Network operators and network operators need to use the network data flow to carry out engineering application issues such as network management, network planning and network anomaly detection, and carry out tasks such as traffic engineering, network traffic anomaly detection, congestion control, etc., so as to better ensure the service quality and normal operation of the IP network.

[0003] In order to observe the characteristics and flow of network traffic from the perspective of the entire network, it is necessary to use the origin-destination (OD) flow to count and describe the network traffic. OD flow is the abbreviation of OD traffic intensity, that is, the traffic size from the source node to the destination node in the network. The traffic size can be in bytes, packets or the number of flows. OD flow can more intuitively reflect the basic properties of the network, is the basis of global traffic analysis, and is also the main input parameter of traffic engineering. The matrix that records the traffic value transmitted between all source-destination node pairs of the measured network is called the Traffic Matrix (TM). The traffic matrix describes the distribution of network traffic between each OD pair and is widely used in traffic engineering, full network anomaly detection and other application problems.

[0004] Although OD traffic can essentially reflect the actual status of the network, it is difficult to directly measure the OD flow in the network. Although there are some measurement software or hardware that can directly measure the OD flow of a specific five-tuple (called flow flow), such as Cisco's Netflow tool. Netflow can be built into a router, sampling the data packets passing through the router, and then obtaining its source IP address and destination IP address by analyzing the packet protocol header information, thereby obtaining the OD flow of a specific IP prefix. However, Netflow only supports routers produced by Cisco, and obtaining OD flow through Netflow is too "expensive" and consumes a lot of router resources.

[0005] In contrast to OD flows, the traffic size through network links is very easy to obtain. The traffic size of a link, i.e., link load, can be easily obtained through the Simple Network Management Protocol (SNMP). Since link load is easy to obtain, the current work attempts to estimate the traffic matrix through the traffic of links in the network, i.e., mapping from link traffic to OD flows.

[0006] However, the traffic carried by a link is the sum of the OD traffic routed through it, and the OD flow is proportional to the link load. Through the linear equation Association, vector Indicates the network The load of the link, For the network OD flow, is the load measurement error, The matrix is ​​also called the routing matrix, which means whether a link is passed by the corresponding OD flow. Generally speaking, if there is no multi-path routing in the network, the first The link is The representation of the OD flow passing through is equal to 1, otherwise ; If there are multi-path routes in the network, then The value of OD flow passes through the The traffic ratio of the link.

[0007] Since the number of OD flows is much larger than the number of links Solving the OD flow in the linear equation is a typical underdetermined problem. The traditional solution method is to solve it based on the assumptions of the model, that is, to make certain assumptions about the underlying distribution of the traffic, such as a network traffic estimation method provided by the Chinese invention patent with publication number CN102325090A, and an end-to-end network traffic reconstruction method based on compressed sensing in a dynamic network provided by the Chinese invention patent with publication number CN102724078A. Yin Zhang et al. used link load and network routing configuration information to estimate OD flow in the paper "Fast accurate computation of large-scale IP traffic matrices from link loads", and Xiao-Bo Fan et al. used the sparse characteristics of traffic for reconstruction in the paper "Sparse representation for network traffic recovery".

[0008] The estimation accuracy of the above existing technologies often depends on the accuracy of the underlying modeling of network traffic and also on the setting of the model's default parameters. The model comes from people's prior knowledge of traffic. No model can fully and accurately characterize the characteristics of OD flow, which inevitably leads to estimation errors. Summary of the invention

[0009] The purpose of the present invention is to provide an OD flow estimation method and system based on spatiotemporal network representation, which completely adopts a data-driven approach to solve the problem. By learning the essential characteristics and prior knowledge of network traffic, using deep neural networks such as encoders and decoders to characterize their own space, an algorithm is designed to achieve efficient and high-precision estimation of OD flow and abnormal traffic, so as to solve the problem of estimation errors caused by existing solution methods.

[0010] The embodiment of the present invention is implemented by the following technical solution: an OD flow estimation method based on spatiotemporal network representation, comprising the following steps:

[0011] Obtain network traffic data and build a traffic dataset;

[0012] Constructing a traffic self-representation spatiotemporal network based on a spatiotemporal variational autoencoder and using the traffic dataset for training, wherein the encoding part of the spatiotemporal variational autoencoder introduces CNN and LSTM to express the spatiotemporal correlation characteristics in network traffic, and the decoding part of the spatiotemporal variational autoencoder is used as a generator;

[0013] The link load data is input into a generator of a trained flow self-characterization spatiotemporal network, the OD flow matrix of the target network is solved using the link load data in the representation space of the generator, and an estimated value of the OD flow matrix is ​​output.

[0014] According to a preferred embodiment, the network traffic data is public network traffic data and / or historical network traffic data from different networks.

[0015] According to a preferred embodiment, the CNN includes K convolutional layers and one fully connected layer, where K is a positive integer greater than or equal to 1.

[0016] According to a preferred embodiment, the CNN processing of the input signal includes: at each time step, The OD flow and its neighboring OD flows are normalized to obtain network traffic data. , treating it as a picture, As the input signal of the convolution layer and pass it to K convolution layers for convolution operation, the output ,Will Input the fully connected layer to get the state vector , with the state vector As the input of LSTM, Indicates time steps, represents the dimension of the image obtained after normalization. The strip OD stream is located at the center of the image;

[0017] The LSTM state vector The processing process includes: at each time step, receiving the input state vector And compare it with the hidden feature vector of the previous time step Passed together to the LSTM unit, according to the input state vector and the hidden feature vector at the previous time step Calculate the forget gate and / or input gate, update the memory cell state at the current moment according to the calculation results of the forget gate and / or input gate, and update the hidden feature vector of the memory cell state at the current time step and the previous time step according to the calculation results of the forget gate and / or input gate. Calculate the output gate to get the hidden feature vector of the current time step As the input of the next time step, after After time steps, the hidden feature vector of the current time step is output.

[0018] According to a preferred embodiment, at each time step, the input of the LSTM unit also includes a vector for encoding context features other than traffic. .

[0019] According to a preferred embodiment, the network decoding part of the traffic self-characterization spatiotemporal network adopts a fully connected layer.

[0020] According to a preferred embodiment, the lasso optimization formula for solving the target network OD flow matrix is ​​as follows:

[0021]

[0022] In the above formula, represents the estimated normal OD flow, Represents the representation vector of the OD flow in the trained traffic self-representation spatiotemporal network, The decoding part of the trained traffic self-representation spatiotemporal network, represents the estimated abnormal flow component, Indicates the mutation point in the flow, represents the error term excluding flow anomaly, Indicates the link load, , Indicates the number of links in the network, Indicates the measurement and The weight of Indicates the sparseness of traffic anomalies.

[0023] The present invention also provides an OD flow estimation system based on spatiotemporal network representation, comprising:

[0024] A data acquisition module, used to acquire network traffic data and construct a traffic data set;

[0025] A traffic self-characterization spatiotemporal network construction module is used to construct a traffic self-characterization spatiotemporal network based on a spatiotemporal variational autoencoder and use the traffic data set for training, wherein the encoding part of the spatiotemporal variational autoencoder introduces CNN and LSTM to express the spatiotemporal correlation characteristics in network traffic, and the decoding part of the spatiotemporal variational autoencoder is used as a generator;

[0026] A solution module is used to input link load data into a generator of a trained flow self-characterization spatiotemporal network, solve the OD flow matrix of the target network using the link load data in the representation space of the generator, and output an estimated value of the OD flow matrix.

[0027] The technical solution of an OD flow estimation method and system based on spatiotemporal network characterization provided by the present invention has at least the following advantages and beneficial effects: the present invention completely adopts a data-driven approach to solve, fully utilizes the spatiotemporal characteristics in the network traffic matrix, learns the essential characteristics and prior knowledge of network traffic, uses deep neural networks such as encoders and decoders to characterize its own space, and designs algorithms to achieve efficient and high-precision estimation of OD flow and abnormal traffic. The efficient and high-precision OD flow estimation can be used for tasks such as link planning and traffic prediction, and the efficient and high-precision abnormal traffic can provide evidence for discovering network failures or network attacks and other events. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of the overall process of an OD flow estimation method based on spatiotemporal network representation provided in Example 1 of the present invention;

[0029] Figure 2 A flow chart of the OD flow estimation method provided in Example 1 of the present invention;

[0030] Figure 3 It is the basic framework of variational autoencoder;

[0031] Figure 4 The encoding part of the spatiotemporal variational autoencoder provided in Embodiment 1 of the present invention;

[0032] Figure 5 A schematic diagram of the amplitude of abnormal components in the OD flow provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0034] Example 1

[0035] Figure 1 A schematic diagram of a flow chart of an OD flow estimation method based on spatiotemporal network representation provided by an embodiment of the present invention, Figure 2 Flow chart of the OD flow estimation method provided by the embodiment of the present invention. Figure 1 and Figure 2 As shown, the OD flow estimation method includes the following steps:

[0036] Step 1: Traffic dataset construction

[0037] In this embodiment, a large amount of public network traffic data and historical network traffic data of different topology networks and business scenarios are collected; public network traffic data such as Abilene, NET82, GÉANT, Sprint and other public network traffic data can be collected by Netflow deployed in network nodes; historical network traffic data is the historical OD flow data of the network where the estimated traffic is located.

[0038] Furthermore, the collected public network traffic data and historical network traffic data are fused to obtain a traffic data set, which is used as a training data set for subsequent use in traffic self-characterization spatiotemporal network training.

[0039] Step 2: Traffic Self-Characterization

[0040] In a possible implementation manner of the embodiment of the present invention, a generative model, such as a variational autoencoder (VAE) and a generative adversarial network (GAN), is used to characterize the generation space of the original data.

[0041] It should be noted that since the main purpose of the generative adversarial network is to generate new samples, and the variational autoencoder can also model the distribution of data and learn the implicit representation of the data, this embodiment uses the variational autoencoder to design the flow self-representation spatiotemporal network, and will not go into details about the generative adversarial network.

[0042] As an unsupervised model, this embodiment uses a variational autoencoder for automatic feature learning, serving the OD flow solution in the generator representation space. Figure 3 As shown, the variational autoencoder includes an encoding part and a decoding part. For input data, To output data; in this embodiment, after the OD stream is input into the variational autoencoder, the variational autoencoder generates a potential probability distribution of the OD stream through the encoding part, usually in the form of a Gaussian distribution, and then randomly samples from the distribution to obtain a continuous and complete latent space, and then reconstructs the original OD stream through the decoding part.

[0043] Considering that the decoding part of the variational autoencoder, as a generator representation, makes an independent and identically distributed assumption in the generation process of OD flow estimation, it is not suitable for directly representing a data set with strong correlation such as network traffic. Based on this, this embodiment also provides another implementation method.

[0044] In another possible implementation of this embodiment, a temporal-spatial variational autoencoder (TS-VAE) is designed as a traffic self-characterization spatiotemporal network, and the network spatiotemporal characteristics are fully utilized in the network coding part to accelerate the network training.

[0045] Furthermore, this embodiment introduces a convolutional neural network (CNN) to capture spatial correlation and a long short-term memory (LSTM) to model temporal dependency in the encoding part of the spatiotemporal variational autoencoder, combining the advantages of CNN and LSTM to capture the spatiotemporal correlation in the OD flow, so that the spatiotemporal correlation can be fully utilized to accelerate the network training when performing traffic self-characterization. Figure 4 As shown, the matrix X To represent the value of OD flow at different times, where the matrix X Each column is a vector formed by stacking TM matrices at a certain moment, and each row represents the value of a certain OD flow at different moments.

[0046] Furthermore, in this embodiment, CNN includes K convolutional layers Conv to perform convolution operations on input signals and a fully connected layer FC to obtain a low-dimensional representation, where K is a positive integer greater than or equal to 1.

[0047] The CNN processing process of the input signal includes: at each time step The OD flow and its neighboring OD flows are normalized to obtain network traffic data. , treating it as a picture, As the input signal of the convolution layer and pass it to K convolution layers for convolution operation, the output ,Will Input the fully connected layer to get the state vector , with the state vector As the input of LSTM, Indicates time steps, represents the dimension of the image obtained after normalization, controlling the maximum radiation range of spatial dependence. The strip OD stream is located in the center of the image.

[0048] LSTM learns time correlation by recursively applying a transformation function to the input state vector. Specifically in this embodiment, the LSTM applies a transformation function to the state vector The processing process includes: at each time step, receiving the input state vector And compare it with the hidden feature vector of the previous time step Passed together to the LSTM unit; according to the input state vector and the hidden feature vector at the previous time step Calculating the forget gate and / or input gate , , , represents the hidden state vector, Represents the sigmoid activation function; updates the current memory unit according to the calculation results of the forget gate and / or input gate state, , Represents the state of the memory cell at the previous time step; based on the state of the memory cell at the current time step and the hidden feature vector of the previous time step Compute output gate , , get the hidden feature vector of the current time step As the input for the next time step, , after After time steps, the hidden feature vector of the current time step is output, where represents the Hadamard product, represents the hyperbolic tangent function, both of which operate element by element. , , are all learnable parameters, represents the weight matrix, Represents the bias vector.

[0049] Furthermore, since each topology is different when performing traffic characterization, in addition to the state vector In addition, the input of the LSTM unit also includes a vector for encoding contextual features other than traffic. , that is, in the LSTM unit, , Operator representing vector concatenation.

[0050] Step 3: OD flow matrix estimation

[0051] In a possible implementation of this embodiment, the link load data is input into the generator of the trained flow self-representation spatiotemporal network, and the link load data is used in the representation space of the generator to solve the OD flow matrix of the target network, that is, to solve the linear equation , output the estimated value of the OD flow matrix.

[0052] When designing the algorithm, we consider that the network OD flow usually has traffic anomalies and measurement errors, that is, in reality, due to network attacks or network configuration errors, network traffic anomalies will occur, and this anomaly is manifested as a sudden change at certain time points. Based on this, the lasso optimization formula for solving the target network OD flow matrix in this example is as follows:

[0053]

[0054] In the above formula, The decoding part of the trained traffic self-representation spatiotemporal network, Represents the estimated normal OD flow. Efficient and high-precision OD flow estimation can be used for tasks such as link planning and traffic prediction. Represents the representation vector of the OD flow in the trained traffic self-representation spatiotemporal network, Indicates the estimated abnormal traffic volume. High-efficiency and high-precision abnormal traffic can provide evidence for discovering network failures or network attacks. Indicates the mutation point in the flow, represents the error term excluding flow anomaly, Indicates the link load, , Indicates the number of links in the network, Indicates the measurement and The weight of Indicates the sparseness of traffic anomalies.

[0055] See also Figure 5 As shown, Figure 5 The abnormal component amplitude of the OD flow and abnormal flow of the Abilene network is shown by using the OD flow estimation method provided in this embodiment to estimate the OD flow and abnormal flow efficiently and accurately. The z-axis is the normalized amplitude. Figure 5It can be seen that at a few time points, the OD flow will suddenly surge, which manifests as abnormal traffic.

[0056] In summary, the present invention completely adopts a data-driven approach to solve the problem, fully utilizes the spatiotemporal characteristics in the network traffic matrix, learns the essential characteristics and prior knowledge of network traffic, uses deep neural networks such as encoders and decoders to characterize its own space, and designs algorithms to achieve efficient and high-precision estimation of OD flow and abnormal traffic. The efficient and high-precision OD flow estimation can be used for tasks such as link planning and traffic prediction, and the efficient and high-precision abnormal traffic can provide evidence for discovering network failures or network attacks.

[0057] Example 2

[0058] The embodiment of the present invention is based on the technical solution provided in Example 1, and provides an OD flow estimation system based on spatiotemporal network characterization, including: a data acquisition module, used to acquire network traffic data and construct a traffic data set; a traffic self-characterization spatiotemporal network construction module, used to construct a traffic self-characterization spatiotemporal network based on a spatiotemporal variational autoencoder and use the traffic data set for training, wherein the encoding part of the spatiotemporal variational autoencoder introduces CNN and LSTM to express the spatiotemporal correlation characteristics in the network traffic, and the decoding part of the spatiotemporal variational autoencoder is used as a generator; a solution module, used to input the link load data into the generator of the trained traffic self-characterization spatiotemporal network, use the link load data in the representation space of the generator to solve the OD flow matrix of the target network, and output the estimated value of the OD flow matrix.

[0059] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An OD flow estimation method based on spatiotemporal network representation, characterized in that: The following steps are involved: Obtain network traffic data and build a traffic dataset; Constructing a traffic self-representation spatiotemporal network based on a spatiotemporal variational autoencoder and using the traffic dataset for training, wherein the encoding part of the spatiotemporal variational autoencoder introduces CNN and LSTM to express the spatiotemporal correlation characteristics in network traffic, and the decoding part of the spatiotemporal variational autoencoder is used as a generator; Inputting the link load data into a generator of a trained flow self-representation spatiotemporal network, solving the OD flow matrix of the target network using the link load data in the representation space of the generator, and outputting an estimated value of the OD flow matrix; The CNN includes K convolutional layers and one fully connected layer, where K is a positive integer greater than or equal to 1; The CNN processing process of the input signal includes: at each time step The OD flow and its neighboring OD flows are normalized to obtain network traffic data. , treating it as a picture, As the input signal of the convolution layer and pass it to K convolution layers for convolution operation, the output ,Will Input the fully connected layer to get the state vector , with the state vector As the input of LSTM, Indicates time steps, represents the dimension of the image obtained after normalization. The strip OD stream is located at the center of the image; The LSTM state vector The processing process includes: at each time step, receiving the input state vector And compare it with the hidden feature vector of the previous time step Passed together to the LSTM unit, according to the input state vector and the hidden feature vector at the previous time step Calculate the forget gate and / or input gate, update the memory cell state at the current moment according to the calculation results of the forget gate and / or input gate, and update the hidden feature vector of the memory cell state at the current time step and the previous time step according to the calculation results of the forget gate and / or input gate. Calculate the output gate to get the hidden feature vector of the current time step As the input of the next time step, after After time steps, the hidden feature vector of the current time step is output.

2. The OD flow estimation method based on spatiotemporal network representation according to claim 1, characterized in that: The network traffic data is public network traffic data and / or historical network traffic data from different networks.

3. The OD flow estimation method based on spatiotemporal network representation according to claim 1, characterized in that: At each time step, the input of the LSTM unit also includes a vector for encoding contextual features other than traffic. .

4. The OD flow estimation method based on spatiotemporal network representation according to claim 1, characterized in that: The network decoding part of the traffic self-representation spatiotemporal network adopts a fully connected layer.

5. The OD flow estimation method based on spatiotemporal network representation according to claim 1, characterized in that: The lasso optimization formula for solving the target network OD flow matrix is ​​as follows: In the above formula, represents the estimated normal OD flow, Represents the representation vector of the OD flow in the trained traffic self-representation spatiotemporal network, The decoding part of the trained traffic self-representation spatiotemporal network, represents the estimated abnormal flow component, Indicates the mutation point in the flow, represents the error term excluding flow anomaly, Indicates the link load, , Indicates the number of links in the network. Indicates the measurement and The weight of Indicates the sparseness of traffic anomalies.

6. An OD flow estimation system based on spatiotemporal network representation, characterized in that: include: A data acquisition module, used to acquire network traffic data and construct a traffic data set; A traffic self-characterization spatiotemporal network construction module is used to construct a traffic self-characterization spatiotemporal network based on a spatiotemporal variational autoencoder and use the traffic data set for training, wherein the encoding part of the spatiotemporal variational autoencoder introduces CNN and LSTM to express the spatiotemporal correlation characteristics in network traffic, and the decoding part of the spatiotemporal variational autoencoder is used as a generator; A solution module, used for inputting link load data into a generator of a trained flow self-representation spatiotemporal network, solving an OD flow matrix of a target network using the link load data in a representation space of the generator, and outputting an estimated value of the OD flow matrix; The CNN includes K convolutional layers and one fully connected layer, where K is a positive integer greater than or equal to 1; The CNN processing process of the input signal includes: at each time step The OD flow and its neighboring OD flows are normalized to obtain network traffic data. , treating it as a picture, As the input signal of the convolution layer and pass it to K convolution layers for convolution operation, the output ,Will Input the fully connected layer to get the state vector , with the state vector As the input of LSTM, Indicates time steps, represents the dimension of the image obtained after normalization. The strip OD stream is located at the center of the image; The LSTM state vector The processing process includes: at each time step, receiving the input state vector And compare it with the hidden feature vector of the previous time step Passed together to the LSTM unit, according to the input state vector and the hidden feature vector at the previous time step Calculate the forget gate and / or input gate, update the memory cell state at the current moment according to the calculation results of the forget gate and / or input gate, and update the hidden feature vector of the memory cell state at the current time step and the previous time step according to the calculation results of the forget gate and / or input gate. Calculate the output gate to get the hidden feature vector of the current time step As the input of the next time step, after After time steps, the hidden feature vector of the current time step is output.

Images