A method for predicting related network traffic based on multi-head attention mechanism
The multi-head attention mechanism enhances network traffic prediction by linking correlated data sets, improving accuracy in complex scenarios by extracting and utilizing correlation features.
Patent Information
- Application Number
- CN202210841515.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-18
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2042-07-18
AI Technical Summary
Existing network traffic prediction models fail to effectively utilize the correlation between different groups of network traffic, leading to inaccurate predictions, especially in scenarios with unstable and complex network traffic patterns.
A method utilizing a multi-head attention mechanism to link related network traffic data sets, incorporating a multi-head attention mechanism to extract correlation features and predict traffic, involving two encoders, a decoder, and a multi-head attention mechanism to enhance prediction accuracy.
Improves prediction accuracy by leveraging correlations between network traffic data sets, achieving higher accuracy through flexible neural network configurations tailored to specific scenarios.
Smart Images
Figure CN115146732B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of network traffic prediction, and specifically relates to a related network traffic prediction method based on a multi-head attention mechanism. Background Art
[0002] In recent years, Internet traffic has grown rapidly, which poses a challenge to the reasonable allocation of network resources. In order to prevent data transmission conflicts, reduce network latency, and ensure network quality, a suitable network traffic prediction algorithm is needed to help network operators and other related units to reasonably schedule and arrange network resources.
[0003] According to the research of domestic and foreign scholars in recent years, network traffic prediction algorithms can be mainly divided into the following two types of models: (1) Linear prediction models; such as the autoregressive moving average model (ARMA) and the Markov model (Markov). When analyzing network traffic, scholars found that it has correlation and self-similarity properties, and proposed linear prediction models based on this. They have the advantages of few parameters and simple calculation. Such models work better when the network traffic is not large and relatively stable. However, with the development of the Internet, more and more network applications, and more and more unstable network traffic, such models are gradually no longer applicable. (2) Nonlinear prediction models; this type of model can also be roughly divided into two parts, neural network algorithms and other machine learning algorithms, such as recurrent neural networks (RNN) and gray prediction model algorithms (GM). The GM model can analyze the nonlinear characteristics of network traffic data, but its prediction effect on irregular data is poor. Neural networks have attracted widespread attention from scholars because of their self-learning and powerful functions, and are distinguished from other machine learning algorithms. The RNN algorithm can extract the time characteristics of data, so it has an advantage in dealing with time series problems. In recent years, the RNN algorithm and its variants have been applied to time series problems in various fields.
[0004] Currently, in the field of network traffic prediction, most studies only consider the time correlation of network traffic itself. In practice, different groups of network traffic in the same scenario often have a certain correlation, such as the network traffic of different optical network units in a wireless optical access network, but this correlation has not been well utilized. Summary of the invention
[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for predicting related network traffic based on a multi-head attention mechanism, which links the related network traffic with a multi-head attention mechanism, extracts correlation features, and uses them to predict traffic, aiming to make the prediction results more accurate.
[0006] The objective of the present invention is achieved through the following technical solutions:
[0007] A related network traffic prediction method based on a multi-head attention mechanism, comprising:
[0008] Load a set of network traffic data to be predicted as the main prediction data, calculate the correlation coefficient between the data and other sets of network traffic data in the same network to determine the correlation between the two sets of data, and select a set of other sets of network traffic data with the greatest correlation with the main prediction data as auxiliary prediction data;
[0009] Normalize the main prediction data and auxiliary prediction data respectively;
[0010] Use the data of m steps before time T in the main prediction data and the data of n steps before time T in the auxiliary prediction data to determine the input sequence of the main prediction data and the input sequence of the auxiliary prediction data, and use the relevant network traffic prediction model based on the multi-head attention mechanism to predict the data of p steps from time T in the main prediction data;
[0011] The related network traffic prediction model based on the multi-head attention mechanism includes two encoders HEAD1 and HEAD2, a decoder and a multi-head attention mechanism; HEAD1 inputs data in the main prediction data input sequence, HEAD2 inputs data in the auxiliary prediction data input sequence, and the output of the decoder is the prediction result; the multi-head attention mechanism connects the encoder and the decoder;
[0012] The output results of HEAD1 and HEAD2 are input into the multi-head attention mechanism, and are weighted and added with the attention weights in the multi-head attention mechanism to serve as the input of the decoder at each moment; the attention weights of the multi-head attention mechanism The calculation formula is as follows:
[0013]
[0014]
[0015] in Indicates the i-th prediction result and the j-th output result of HEADnum The weight of Indicates the i-th prediction result and the k-th output result of HEAD1 The weight of Represents the i-th prediction result and the g-th output result of HEAD2 The weight of After normalization, we get the normalized weights represents the hidden state of the LSTM model in the decoder at time i-1, W e , W e1 , W e2 and b eThese are parameters that require iterative learning; num = 1, 2;
[0016] Based on the output of the encoder and the attention weights of the multi-head attention mechanism, the output of the multi-head attention mechanism is calculated as follows:
[0017]
[0018] Among them C i is the i-th output of the multi-head attention mechanism, i.e., the i-th input of the decoder, Indicates the i-th prediction result and the k-th output result of HEAD1 The normalized weight of Represents the i-th prediction result and the g-th output result of HEAD2 The normalized weights are then used, and then two fully connected layers are used to reduce the results of each step to one dimension to obtain the final prediction result.
[0019] Furthermore, the method uses the data m steps before time T in the main prediction data and the data n steps before time T in the auxiliary prediction data, and uses the relevant network traffic prediction model based on the multi-head attention mechanism to predict the data p steps from time T in the main prediction data, including:
[0020] A sliding window method is used to divide the main prediction data into a group of m-length main prediction data input sequences and a group of p-length prediction result sequences by m and p respectively, and to divide the auxiliary prediction data into a group of n-length auxiliary prediction data input sequences by n, and the three groups of sequences correspond to each other; the first m-length data in the main prediction data input sequence and the first n-length data in the auxiliary prediction data input sequence are used together to predict the first p-length data in the prediction result sequence.
[0021] Furthermore, the neural network models in the encoder and decoder both adopt the long short-term memory network LSTM model, and the state expression is as follows:
[0022]
[0023]
[0024]
[0025]
[0026] h t =o t ⊙tanh(s t )
[0027] where i t 、f tand t Respectively represent the functions corresponding to the input gate, forget gate and output gate, s t is the process parameter, is the current input, h t and h t-1 Respectively represent the current hidden state and the previous hidden state, h t-1 and The splicing of W f , W i , W o and W s Represent the input weight matrix in the input gate, forget gate, output gate and memory unit, respectively, b f , b i , b o and b s They represent the bias in the input gate, forget gate, output gate and memory unit respectively, σ represents the activation function, which is set to the sigmoid function, and ⊙ represents element-wise multiplication.
[0028] Furthermore, when the data in the main prediction data input sequence and the data in the auxiliary prediction data input sequence are greater than the preset threshold, the number of layers and the number of hidden neurons in the LSTM model in the encoder are increased; if the accuracy of the prediction results of the decoder does not meet the preset requirements, the number of layers and the number of hidden neurons in the LSTM model in the decoder are increased.
[0029] Furthermore, the input sequence length of HEAD1 is m=15, the number of layers of the LSTM model in HEAD1 is 2, and the number of hidden layer neurons is 128; the input sequence length of HEAD2 is n=5, the number of layers of the LSTM model in HEAD2 is 2, and the number of hidden layer neurons is 128; the output sequence length of the decoder is p=5, the number of network layers is 1, and the number of hidden layer neurons is 128.
[0030] Further, the first 90% of the data in the main prediction data input sequence, the prediction result sequence, and the auxiliary prediction data input sequence are respectively taken as training sets, and the remaining parts are taken as test sets;
[0031] Input the data in the training set into the traffic prediction model, perform iterative training, and obtain a trained model; set the batch training data size, divide the data in the training set into several batches, set the learning rate size and the number of iterations, input the main prediction data and auxiliary prediction data into the corresponding encoders in batches, calculate the loss function after obtaining the prediction results, use the Adam algorithm for iterative optimization, and determine whether the predetermined number of iterations has been reached. If not, repeat the iterative calculation; if satisfied, save the model as a trained model;
[0032] The prediction accuracy of the trained model is tested using the test set. When the preset accuracy is met, the model is saved as the final prediction model.
[0033] Compared with the prior art, the present invention has the following technical features:
[0034] The present invention utilizes the correlation between different network flows, extracts corresponding features with corresponding neural networks, connects encoders and decoders with a multi-head attention mechanism, and combines them with time features for prediction to improve the accuracy of flow prediction. Different encoders and decoders can set different neural networks or different parameters according to specific circumstances, which increases the flexibility of the model when used. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is the model principle diagram of the present invention;
[0036] Figure 2 A network traffic data trend chart used in the embodiment;
[0037] Figure 3 This is a comparison diagram of neural network model training in the embodiment;
[0038] Figure 4 This is a result diagram of network traffic prediction in the embodiment. DETAILED DESCRIPTION
[0039] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0040] This embodiment provides a method for predicting related network traffic based on a multi-head attention mechanism. The principle is as follows: Figure 1 As shown, the network traffic prediction method includes the following steps:
[0041] S1, first obtain a set of main prediction data and another set of auxiliary prediction data with certain correlation, such as Figure 2 As shown, two sets of original network traffic data are preprocessed to adjust them into data sequences that meet the input of the model and divided into training data and test data. Specifically:
[0042] S1.1, load a set of network traffic data that needs to be predicted as the main prediction data, calculate the Spearman correlation coefficient between the data and other sets of network traffic data in the same network to determine the correlation between the two sets of data. The closer the absolute value of the Spearman correlation coefficient is to 1, the more correlated the two sets of data are. Select a set of other sets of network traffic data with the greatest correlation with the main prediction data as auxiliary prediction data, and store the main prediction data and auxiliary prediction data locally.
[0043] S1.2, the main prediction data and auxiliary prediction data are one-dimensional data sequences, and the maximum value x of each set of data is obtained. max and the minimum value x min , and perform min-max normalization on each.
[0044] S1.3, use the data m steps before time T in the main prediction data and the data n steps before time T in the auxiliary prediction data to predict the data p steps from time T in the main prediction data:
[0045] The sliding window method is used to divide the main prediction data into a main prediction data input sequence consisting of multiple m-length data and a prediction result sequence consisting of multiple p-length data by using m and p respectively. Then, the same method is used to divide the auxiliary prediction data into a auxiliary prediction data input sequence consisting of multiple n-length data by using n. The three sequences correspond to each other. For example, when m is greater than n, the first m-length data in the main prediction data input sequence is used. and the first n-length data of the auxiliary prediction data input sequence Commonly used to predict the first p-length data in the prediction result sequence {Y m+1 ,Y m+2 ,…,Y m+p},in Represents the main forecast data at time t, Represents the auxiliary prediction data at time t, Y t Represents the prediction result data at time t.
[0046] S1.4, take the first 90% of the data in the three sets of number series obtained in S1.3 as training sets, and the rest as test sets.
[0047] The experimental data in this embodiment is obtained by simulating the ON / OFF model. The ON / OFF model used is composed of 16 ON / OFF sources superimposed. The ON and OFF time of each source satisfies the Pareto distribution. The obtained data is superimposed every 50 data to finally obtain the main traffic data. The data set contains 444443 data. The self-similarity coefficient H of this group of data is calculated to be 0.8864, which meets the network traffic characteristics. The auxiliary prediction data is obtained by changing the sampling superposition method. The spearman correlation coefficient of the two groups of data is 0.85, which is highly correlated and meets the experimental requirements. The trend chart is as follows Figure 2 shown.
[0048] Experimental environment: operating system Windows 10, graphics card NVIDIA GeForce RTX 3090, algorithm framework Pytorch.
[0049] S2, builds a related network traffic prediction model based on the multi-head attention mechanism; the model includes two encoders, a decoder and a multi-head attention mechanism. The two encoders correspond to Figure 1 HEAD1 and HEAD2 in the code, HEAD1 inputs the data in the main prediction data input sequence, HEAD2 inputs the data in the auxiliary prediction data input sequence, and the output of the decoder is the prediction result, that is, the data in the prediction result sequence; the multi-head attention mechanism connects the encoder and the decoder; specifically:
[0050] The neural network in the encoder is used to extract feature information of the input data, and the neural network in the decoder is used to analyze and calculate the prediction results from the features. In this embodiment, the neural network models in the encoder and the decoder both adopt the long short-term memory network (LSTM) model, and the state expression is as follows:
[0051]
[0052]
[0053]
[0054]
[0055] h t =o t ⊙tanh(s t )
[0056] where i t 、f t and tThey represent the functions corresponding to the input gate, forget gate, and output gate, respectively. t is the process parameter, is the current input, h t and h t-1 Respectively represent the current hidden state and the previous hidden state, h t-1 and The splicing of W f , W i , W o , W s , b f , b i , b o and b s are all parameters that need to be learned iteratively, W f , W i , W o and W s Represent the input weight matrix in the input gate, forget gate, output gate and memory unit, respectively, b f , b i , b o and b s They represent the bias in the input gate, forget gate, output gate and memory unit respectively, σ represents the activation function, which is set to the sigmoid function, and ⊙ represents element-wise multiplication.
[0057] The relevant parameters of the LSTM model include the number of hidden layer neurons, the number of network layers, the input data dimension, etc. The LSTM model parameters in different encoders and decoders can be set differently according to the situation. For example, if the input data steps are long (greater than the preset threshold) or more complex, the number of layers and hidden layer neurons of the LSTM model in the encoder can be increased. If the prediction data results are not accurate enough, the number of layers and hidden layer neurons of the LSTM model in the decoder can be increased.
[0058] In this embodiment, the input sequence length of HEAD1 is m=15, the number of layers of the LSTM model in HEAD1 is 2, and the number of hidden layer neurons is 128; the input sequence length of HEAD2 is n=5, the number of layers of the LSTM model in HEAD2 is 2, and the number of hidden layer neurons is 128. The output sequence length of the decoder is p=5, the number of network layers is 1, and the number of hidden layer neurons is 128.
[0059] and They are the output results of HEAD1 and HEAD2 respectively. The output results are input into the multi-head attention mechanism, and are weighted and added with the attention weight in the multi-head attention mechanism as the input of the decoder at each moment; the attention weight of the multi-head attention mechanism The calculation formula is as follows:
[0060]
[0061]
[0062] in Indicates the i-th prediction result and the j-th output result of HEADnum The weight of Indicates the i-th prediction result and the k-th output result of HEAD1 The weight of Represents the i-th prediction result and the g-th output result of HEAD2 The weight of After normalization, we get the normalized weights represents the hidden state of the LSTM model in the decoder at time i-1, W e , W e1 , W e2 and b e These are parameters that require iterative learning; num=1,2.
[0063] Based on the output of the encoder and the attention weights of the multi-head attention mechanism, the output of the multi-head attention mechanism is calculated as follows:
[0064]
[0065] Among them C i is the i-th output of the multi-head attention mechanism, i.e., the i-th input of the decoder, which is obtained by the LSTM model in the decoder based on the input of p steps {C 1 ,C 2 ,…,C p}Calculate the output results of each step in turn, Indicates the i-th prediction result and the k-th output result of HEAD1 The normalized weight of Represents the i-th prediction result and the g-th output result of HEAD2 The normalized weights are finally reduced to one dimension by two fully connected layers. The calculation formula of each fully connected layer is as follows:
[0066] y i =W y A i +b y
[0067] where y i represents the output of the fully connected layer, A i represents the input of the fully connected layer, W y and b y are the parameters that need to be learned, representing the input weight and bias respectively, and finally the prediction result {Y 1 ,Y 2 ,…,Y P}, where Y i Represents the prediction result of the i-th step.
[0068] S3, input the data in the training set in S1 into the traffic prediction model, perform iterative training, and obtain a trained model.
[0069] Set the batch training data size, divide the data in the training set in S1 into several batches, set the learning rate size and number of iterations, input the main prediction data and auxiliary prediction data into the corresponding encoders in batches, calculate the loss function after obtaining the prediction results, use the Adam algorithm for iterative optimization, and determine whether the predetermined number of iterations has been reached. If not, repeat the iterative calculation; if satisfied, save the model as a trained model.
[0070] In this example, the learning rate is set to 0.002, the number of iterations is 300, and the batch size is 512.
[0071] S4, input the data in the test set in S1 into the trained model, and denormalize the model output to obtain the prediction result. The calculation formula is as follows:
[0072] x n =x(x max -x min )+x min
[0073] where x n is the result after denormalization, x max and x min are the maximum and minimum values of x before denormalization, respectively, and x is the data to be denormalized. The denormalization result is the prediction result that meets the normal flow data size.
[0074] In the step S4, the data in the test set in S1 is divided into several batches, and the main prediction data and the auxiliary prediction data are input into the corresponding encoders in batches to obtain prediction results to verify the prediction accuracy of the model; when the accuracy of the model reaches the preset accuracy, the model is saved as the final prediction model; otherwise, the model is retrained.
[0075] This example compares a single-head attention network model with a multi-head attention network model that introduces relevant network traffic. Figure 3 As shown in the figure, the MSE of the multi-head attention network model with relevant network traffic in the training situation is lower. The trained model was tested for prediction. The prediction accuracy of the single-head attention network model was 87.85%. The accuracy of the multi-head attention network model with relevant network traffic was higher, reaching 92.14%. The prediction results are shown in the figure below. Figure 4 shown.
[0076] In summary, the present invention provides a method for predicting related network traffic based on a multi-head attention mechanism. By introducing another set of auxiliary prediction data with certain correlation with the single traffic feature extracted by the traditional model, and combining them by using the multi-head attention mechanism, the prediction accuracy is improved while adding certain flexibility to the model. The present invention can be used in traffic prediction scenarios where there is related network traffic, such as wireless optical access network port traffic prediction.
[0077] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art do not depart from the spirit and scope of the present invention, and should be within the scope of protection of the claims attached to the present invention.
Claims
1. A method for predicting related network traffic based on a multi-head attention mechanism, characterized in that: include: Load a set of network traffic data to be predicted as the main prediction data, calculate the correlation coefficient between the data and other sets of network traffic data in the same network to determine the correlation between the two sets of data, and select a set of other sets of network traffic data with the greatest correlation with the main prediction data as auxiliary prediction data; Normalize the main prediction data and auxiliary prediction data respectively; Use the data of m steps before time T in the main prediction data and the data of n steps before time T in the auxiliary prediction data to determine the input sequence of the main prediction data and the input sequence of the auxiliary prediction data, and use the relevant network traffic prediction model based on the multi-head attention mechanism to predict the data of p steps from time T in the main prediction data; The related network traffic prediction model based on the multi-head attention mechanism includes two encoders HEAD1 and HEAD2, a decoder and a multi-head attention mechanism; HEAD1 inputs data in the main prediction data input sequence, HEAD2 inputs data in the auxiliary prediction data input sequence, and the output of the decoder is the prediction result; the multi-head attention mechanism connects the encoder and the decoder; The output results of HEAD1 and HEAD2 are input into the multi-head attention mechanism, and are weighted and added with the attention weights in the multi-head attention mechanism to serve as the input of the decoder at each moment; the attention weights of the multi-head attention mechanism The calculation formula is as follows: in Indicates the i-th prediction result and the j-th output result of HEADnum The weight of Indicates the i-th prediction result and the k-th output result of HEAD1 The weight of Represents the i-th prediction result and the g-th output result of HEAD2 The weight of After normalization, we get the normalized weights represents the hidden state of the LSTM model in the decoder at time i-1, W e , W e1 , W e2 and b e These are parameters that require iterative learning; num = 1, 2; Based on the output of the encoder and the attention weights of the multi-head attention mechanism, the output of the multi-head attention mechanism is calculated as follows: Among them C i is the i-th output of the multi-head attention mechanism, i.e., the i-th input of the decoder, Indicates the i-th prediction result and the k-th output result of HEAD1 The normalized weight of Represents the i-th prediction result and the g-th output result of HEAD2 The normalized weights are then used, and then two fully connected layers are used to reduce the results of each step to one dimension to obtain the final prediction result.
2. The method for predicting related network traffic based on the multi-head attention mechanism according to claim 1 is characterized in that: The method uses the data m steps before time T in the main prediction data and the data n steps before time T in the auxiliary prediction data, and uses a related network traffic prediction model based on a multi-head attention mechanism to predict the data p steps from time T in the main prediction data, including: A sliding window method is used to divide the main prediction data into a group of m-length main prediction data input sequences and a group of p-length prediction result sequences by m and p respectively, and to divide the auxiliary prediction data into a group of n-length auxiliary prediction data input sequences by n, and the three groups of sequences correspond to each other; the first m-length data in the main prediction data input sequence and the first n-length data in the auxiliary prediction data input sequence are used together to predict the first p-length data in the prediction result sequence.
3. The method for predicting related network traffic based on a multi-head attention mechanism according to claim 1 is characterized in that: The neural network models in the encoder and decoder both use the long short-term memory network LSTM model, and the state expression is as follows: h t =o t ⊙tanh(s t ) where i t 、f t and t Respectively represent the functions corresponding to the input gate, forget gate and output gate, s t is the process parameter, is the current input, h t and h t-1 Respectively represent the current hidden state and the previous hidden state, h t-1 and The splicing of W f , W i , W o and W s Represent the input weight matrix in the input gate, forget gate, output gate and memory unit, respectively, b f , b i , b o and b s They represent the bias in the input gate, forget gate, output gate and memory unit respectively, σ represents the activation function, which is set to the sigmoid function, and ⊙ represents element-wise multiplication.
4. The method for predicting related network traffic based on a multi-head attention mechanism according to claim 1 is characterized in that: When the data in the main prediction data input sequence and the data in the auxiliary prediction data input sequence are greater than the preset threshold, the number of layers and hidden neurons of the LSTM model in the encoder are increased; if the accuracy of the prediction result of the decoder does not meet the preset requirements, the number of layers and hidden neurons of the LSTM model in the decoder are increased.
5. The method for predicting related network traffic based on a multi-head attention mechanism according to claim 1 is characterized in that: The input sequence length of HEAD1 is m=15, the number of layers of the LSTM model in HEAD1 is 2, and the number of hidden layer neurons is 128; the input sequence length of HEAD2 is n=5, the number of layers of the LSTM model in HEAD2 is 2, and the number of hidden layer neurons is 128; the output sequence length of the decoder is p=5, the number of network layers is 1, and the number of hidden layer neurons is 128.
6. The method for predicting related network traffic based on a multi-head attention mechanism according to claim 1 is characterized in that: The first 90% of the data in the main prediction data input sequence, the prediction result sequence, and the auxiliary prediction data input sequence are respectively taken as training sets, and the remaining parts are taken as test sets; Input the data in the training set into the traffic prediction model, perform iterative training, and obtain a trained model; set the batch training data size, divide the data in the training set into several batches, set the learning rate size and the number of iterations, input the main prediction data and auxiliary prediction data into the corresponding encoders in batches, calculate the loss function after obtaining the prediction results, use the Adam algorithm for iterative optimization, and determine whether the predetermined number of iterations has been reached. If not, repeat the iterative calculation; if satisfied, save the model as a trained model; The prediction accuracy of the trained model is tested using the test set. When the preset accuracy is met, the model is saved as the final prediction model.