A network traffic prediction method based on multi-scale dilated convolutional neural network

Abstract

The present invention relates to the field of network traffic prediction technology and discloses a network traffic prediction method using a multi-scale dilated convolutional neural network. The present invention uses five publicly available data sets with large noise and outliers (NTBS, WTSW, AIIA, ISP, and EC) to verify its effectiveness: first, the data is preprocessed and analyzed to make it an input variable; then, a multi-scale dilated convolutional neural network prediction model structure based on a random sparse self-attention mechanism is designed; the processed training set is input into the model to train the network traffic prediction model, and after the training is completed, the model parameters are saved; finally, a test set is input into the trained model to test the model performance. The main innovation of the present invention is the development of a three-branch multi-scale model composed of convolution kernels of different sizes, which makes the entire model more flexible and has lower parameter complexity; and a layer determination mechanism is proposed so that the determination of the layer no longer depends on a specific data set.

Description

Technical Field

[0001] The present invention relates to the technical field of network traffic prediction, and in particular to a network traffic prediction method based on a multi-scale dilated convolutional neural network. Background Art

[0002] With the advancement of electronic communications and the increasing adoption of 5G, network traffic forecasting is becoming a critical issue for many businesses and enterprises. During peak network traffic periods, server loads often exceed their maximum capacity, causing websites and applications to experience slow response times. During off-peak network traffic periods, if services continue to operate according to peak schedules, significant unnecessary waste can occur. Therefore, accurate and efficient network traffic forecasting technology is essential for fully utilizing network resources and effectively improving user experience.

[0003] Early network traffic forecasting relied on traditional statistical methods. For example, for stationary series, the ARIMA model was used; for non-stationary series, smoothing methods were used to convert them into stationary series before applying statistical methods. However, ongoing research has revealed that real-world time series often exhibit nonlinear and non-random characteristics, and the data often contains significant noise. This makes traditional network traffic forecasting models difficult to effectively predict in complex scenarios and special circumstances.

[0004] To address this problem, machine learning and deep learning methods are widely used. Existing network traffic forecasting research primarily focuses on the characteristics of network traffic series. Some researchers have attempted to incorporate external information to describe sudden changes in the series, such as major holidays and weather emergencies. However, the selection of external information is somewhat subjective. Other researchers have proposed decomposing time series into several components and then modeling each component separately.

[0005] However, few researchers have focused on the potential features of different scales in network traffic sequences. Researchers have pointed out that time series can be divided into long-term, medium-term, and short-term patterns. Smaller convolution kernels can extract more specific information and features, while larger convolution kernels can extract more abstract information and features. Therefore, some scholars have proposed the use of multi-scale models based on initial structures, mainly using convolution kernels of different sizes to extract features of different scales from time series. Although multi-scale models are widely used in other fields, there are still two major problems that have not been solved. First, the multi-scale model has too many parameters, so training and inference are not economical in terms of memory usage and communication computing costs, and are prone to overfitting. Second, there is currently no research discussing the specific size of the convolution kernel that should be used for the multi-scale model.

[0006] In recent years, some researchers have applied the self-attention mechanism to time series forecasting, exploiting high-order autocorrelations in time series forecasting and achieving more accurate results. In time series, the self-attention mechanism improves information utilization by learning the underlying correlations between different moments, suppressing invalid information and overweighting useful information. However, the large computational resources required have been criticized. Consequently, many researchers have made contributions to the field of sparsity-based self-attention, reducing computational complexity by introducing various sparsity methods into the attention score matrix.

[0007] The existing solution introduces an initialization module for the initialization-based model. The structure of the initialization module is as follows: Figure 2 As shown in the figure, the basic structure of the initialization module consists of four parts: 1×1 convolution, 3×3 convolution, 5×5 convolution, and 3×3 max pooling. Different convolution sizes are used here to capture different receptive fields, and the final concatenation represents the fusion of features at different scales. The 1×1 convolution before the 3×3 and 5×5 convolutions aims to reduce dimensionality, resulting in a significant reduction in parameters compared to the original architecture. Placing the 1×1 convolution after the 3×3 max pooling, as opposed to placing it before, also reduces the number of parameters. Furthermore, based on the initialization model, Batch Normalization is proposed. This method transforms the input distribution of neurons in each layer into a normal distribution with mean 0 and variance 1, placing it within the sensitive region of the activation function, preventing gradient vanishing and accelerating convergence. Finally, the final fully connected layer is removed and replaced with a global average pooling layer, significantly reducing computational effort.

[0008] In general, existing approaches have the following shortcomings: First, multi-scale models have too many parameters, resulting in high memory usage and computational costs for training and inference, and are prone to overfitting. Second, few studies have considered the differential importance of features across time. Information with different contributions should be assigned different weights to efficiently allocate computing resources. Given limited computing resources, if the network treats all information equally, resources may be wasted. Furthermore, when features that contribute significantly to information are ignored, network traffic prediction performance will be severely impacted. To this end, we propose a network traffic prediction method based on a multi-scale dilated convolutional neural network with a randomized sparsity self-attention mechanism. Summary of the Invention

[0009] (1) Technical problems solved

[0010] To address the shortcomings of existing technologies, the present invention provides a network traffic prediction method using a multi-scale dilated convolutional neural network. This method establishes a three-branch multi-scale model to address the drawbacks of excessive multi-scale parameters and the resulting overfitting during training. A randomized sparse self-attention mechanism is also added to distinguish useful information from network traffic and mitigate the interference of outliers and noise. The disclosed method can effectively extract potential features at different scales and ensure more accurate prediction capabilities.

[0011] (2) Technical solution

[0012] To achieve the above-mentioned objectives, the present invention provides the following technical solution: a network traffic prediction method based on a multi-scale dilated convolutional neural network with random sparsity self-attention mechanism, comprising the following steps:

[0013] Step 1: We obtained five datasets with large noise and outliers, namely NTBS, WTSW, AIIA, ISP, and EC data, sorted and analyzed the data, and used them as input variables;

[0014] Step 2: Divide the sorted data into a training set and a test set with a ratio of 8:2. At the same time, perform a sliding average on the training data set, where the sliding window value is set to 3, that is, the average of the three adjacent numbers in the original data is obtained;

[0015] Step 3: Establish a dilated convolutional neural network prediction model based on random sparse self-attention mechanism;

[0016] Step 4: Input the training set into the dilated convolutional neural network prediction model based on the random sparse self-attention mechanism to train the network traffic prediction model. After the training is completed, save the model parameters.

[0017] Step 5: Load the model parameters to obtain the trained network traffic prediction model, input the test set into the model, and obtain the network traffic prediction value;

[0018] Step 6: Evaluate the prediction performance of the trained network traffic prediction model.

[0019] Preferably, the NTBS and WTSW data in the first step contain multiple time series, while the AIIA, ISP and EC data have only one time series.

[0020] Preferably, the specific content of the third step is:

[0021] S1: Input the input sequence into three branches of the dilated convolutional neural network with convolution kernel sizes of k1, k2 and k3 respectively;

[0022] S2: Determine the number of convolutional layers N1, N2, and N3 through the layer determination mechanism;

[0023] S3: A random sparse self-attention mechanism is inserted into the network between every three dilated convolutional layers to consider the importance differences of different temporal features, and the results of the three branch networks are connected;

[0024] S4: Perform linear transformation on the obtained results to obtain the output sequence.

[0025] Preferably, the model parameters in the fourth step are set as:

[0026] The learning rate is 0.0001, and the Adam optimizer is used to optimize the network model parameters. The loss function adopts the following loss function. The training cycle is 150 rounds. For the datasets ISP, AIIA, and EC, the length of the input sequence is set to 200. For the datasets NTBS and WTSW, the length of the input sequence is set to 504 and 350 respectively, and the length of the output sequence is set to 168 and 47 respectively;

[0027]

[0028]

[0029] in Represents the predicted value, y i,j represents the true value, ω j is the parameter of the model, s2 is the output sequence length, and N is the number of layers of the network.

[0030] Preferably, in the sixth step, MAE, MSE, and MAPE are used to evaluate the performance of the model. The indicators can be defined as follows:

[0031]

[0032]

[0033]

[0034] (3) Beneficial effects

[0035] Compared with the existing technology, the present invention provides a network traffic prediction method using a multi-scale dilated convolutional neural network, which has the following beneficial effects:

[0036] 1. The network traffic prediction method of the multi-scale dilated convolutional neural network, which combines the network traffic prediction model of multi-scale dilated convolution and self-attention mechanism, has achieved better performance on multiple data sets.

[0037] 2. Network traffic prediction method based on a multi-scale dilated convolutional neural network. To simplify the structure of the multi-scale model, we developed a three-branch multi-scale model that introduces dilated convolutions, whose dilation scale increases exponentially with the number of layers. Based on this, we designed a flexible layer determination mechanism that does not rely on a specific dataset, but instead determines the number of network layers based on the size of the convolution kernel and the length of the input sequence. As a result, our three-branch multi-scale model is more flexible and has lower parameter complexity than models based on the initial model.

[0038] 3. The network traffic prediction method of the multi-scale dilated convolutional neural network proposes a random sparse self-attention mechanism and introduces dropout, which can not only distinguish useful information of network traffic, but also help the model alleviate the interference of outliers and noise.

[0039] 4. This multi-scale dilated convolutional neural network network traffic prediction method compares and analyzes the performance of different convolution kernel combinations in multiple data sets, and concludes that the determination of the convolution kernel size should take into account the cycle of the time series, which provides a good reference value for determining the size of the convolution kernel. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the overall structure of the invention;

[0041] Figure 2 Schematic diagram of the initialization module for initializing the model;

[0042] Figure 3 Schematic diagram of the overall structure of the multi-scale dilated convolutional neural network;

[0043] Figure 4 Schematic diagram of the structure of the dilated convolution block;

[0044] Figure 5 Detailed process diagram of the random sparsity self-attention mechanism with an input sequence length of 6; DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0046] See also Figure 1-5 , a network traffic prediction method based on a multi-scale dilated convolutional neural network with random sparse self-attention mechanism, comprising the following steps:

[0047] Step 1: Get data.

[0048] We obtained five datasets with significant noise and outliers: Network Traffic to Base Stations (NTBS), Network Traffic Time Series (WTSW) from Wikipedia pages, AIIA data, ISP data, and EC data. NTBS and WTSW data contain multiple time series, while AIIA, ISP, and EC data only have one. We then preprocessed and analyzed these data to prepare them as input variables.

[0049] Step 2: Select training set and test set;

[0050] The data is divided into a training set and a test set with a ratio of 8:2. The training data set is subjected to a sliding average, where the sliding window value is set to 3, that is, the three adjacent numbers in the original data are averaged to remove noise. When the data is multiple time series, the training set is in the form of: When the data has only one sequence, the training set is in the form of: Where s1 and s2 represent the input and output sequence lengths respectively.

[0051] Step 3: Establish a multi-scale dilated convolutional neural network prediction model based on random sparse self-attention mechanism.

[0052] The overall structure of the model is as follows Figure 3 As shown, the model can be summarized as follows: First, the input sequence is fed into three branches of a dilated convolutional neural network with kernel sizes k1, k2, and k3, respectively, to extract latent features at different scales. A layer determination mechanism then determines the number of convolutional layers, N1, N2, and N3. Furthermore, a random sparse self-attention mechanism is inserted between every three dilated convolutional layers to account for the varying importance of temporal features. The results of the three branches are then concatenated. Finally, a linear transformation is applied to obtain the output sequence. This model can be divided into two parts: a three-branch multi-scale dilated convolutional neural network and a random sparse self-attention mechanism.

[0053] Step 3.1. Dilated Convolutional Neural Network

[0054] Dilated convolution uses a hyperparameter d to skip part of the input, allowing the filter to act on an area larger than the filter's own length. This is equivalent to making the original filter generate a larger filter by adding zeros, thereby increasing the receptive field. The hyperparameter d here changes exponentially with the number of layers. Figure 4This is the module diagram of the dilated convolution block. Its input undergoes dilated convolution, 1×1 convolution, weight normalization, activation function, and dropout. The LeakyReLU function is used as the activation function to prevent excessive dead cells. A 1×1 convolution is added after each dilated convolution layer to perform slight dimensionality reduction. Due to the bottleneck structure after the dilated convolution, the input and output sequences have different numbers of channels. Therefore, a 1×1 convolution is added to the residual connection, and finally the corresponding dimension addition operation is performed to obtain the final value of the dilated convolution block.

[0055] Step 3.2. Random Sparse Self-Attention Mechanism

[0056] The overall structure of the random sparsity self-attention mechanism is as follows Figure 5 As shown. First, the one-dimensional sequence does not meet the requirements of being the input of the self-attention mechanism. In order to solve this problem, we first use the convolution operation to convert the original one channel of the one-dimensional time series into multiple channels, and the length of the sequence obtained before and after the convolution remains equal. Secondly, the sequence is generated by convolution and dropout. If these sequences are connected by rows, the matrices Q and K are obtained. Specific performance: Matrix Q and Matrix K are generated by convolution kernels with a size greater than 1. In order to prevent future information leakage, the last two neurons are removed after the convolution, and the vector V is generated by 1×1 convolution. The convolution is initialized with Gaussian, that is, sampling from a Gaussian distribution with a mean of 0 and a variance of 0 as the initial weight of the convolution. The obtained sequence is then connected to obtain matrices Q and K. Then the Dot product is performed on matrices Q and K to obtain the attention matrix, and the upper triangular part of the corresponding position in the attention matrix has the same value. Therefore, normalization by the Softmax function will inevitably lead to future information leakage. Therefore, in addition to the diagonal positions before normalization, this paper also pays attention to the matrix QK T The upper triangular part of is assigned to -∞, thus avoiding future information leakage. At the same time, some elements of the attention matrix are randomly assigned to 0. On the one hand, the self-attention mechanism has a huge computational cost and memory usage. Using a sparse self-attention mechanism will be more flexible, simpler, and reduce the number of parameters for each update. On the other hand, outliers in noise and time series are crowded everywhere, which will inevitably have a negative impact on the values ​​in the attention matrix, resulting in an erroneous sequence of attention scores. Therefore, dropout will help discard erroneous information. Finally, the random sparse attention matrix is ​​transformed using the activation function so that each value ranges from 0 to 1, and then the result is Dot product with the matrix V to obtain the required attention score, assigning different weights to the input sequence. In summary, That is, the weighted sum of the value vectors (information), where the weight is the distribution of the degree of correlation or similarity between each sample and each piece of information.

[0057] Step 3.3. Layer determination mechanism

[0058] This mechanism only considers the size of the convolution kernel and the length of the input sequence to determine the number of convolution layers. If the input sequence and the output sequence length, the number of neurons in the hidden layer is L, the convolution kernel size is an odd number, equal to k, and the convolution kernel expansion size d = 2 i -1, i represents the i-th layer of the network, and the number of layers of the network is N. Then the lower bound of the receptive field of the topmost neuron in the network is:

[0059]

[0060] In addition, from the symmetry of the receptive field, the range of the receptive field rf is: [RF,min(L,2RF)]

[0061] By inverting the above formula, we can get the number of layers as follows: The values ​​of N1, N2, and N3 can be determined according to this formula.

[0062] Step 3.4. Determine the convolution kernel size

[0063] In multi-scale models, there are few papers that point out how to choose the size of the convolution kernel. In this paper, an important principle for determining the size of the convolution kernel in multi-scale models is to consider the cycle in the time series. Specifically, if a cycle series is equal to T, the sizes of the three convolution kernels should be approximately T. Taking the dataset NTBS as an example, the time series period in NTBS is about 24 hours, so the sizes of the three convolution kernels can be 5, 13, and 23. In other words, the largest convolution kernel should roughly cover a subsequence of a cycle containing long-term features. The other two convolution kernels extract medium-term and short-term features, respectively. However, for the datasets ISP and EC, the choice of size does not follow the principles discussed above. This is because a convolution kernel of size 285 in the network will make training difficult and exhaust the computer's memory. This paper uses convolution kernels of sizes 13, 15, and 17 for the ISP and EC datasets, respectively. This also means that although it is not feasible to consider cycles, three convolution kernels that are close to each other can also be used to independently extract short-term features of different scales. This is the second principle.

[0064] Step 4: Input the training data set established in step 2 into the multi-scale dilated convolutional neural network based on the random sparse self-attention mechanism established in step 3 to train the network traffic prediction model. Set the learning rate to 0.0001 and use the Adam optimizer to optimize the network model parameters. The loss function uses the following Loss function. The training cycle is 150 rounds. Represents the predicted value, y i,j represents the true value, ωj is the model parameter, s2 is the output sequence length, and N is the number of network layers. For the ISP, AIIA, and EC datasets, the input sequence length is set to 200. For the NTBS and WTSW datasets, the input sequence length is set to 504 and 350, respectively, and the output sequence length is set to 168 and 47, respectively. After training is complete, save the model parameters.

[0065]

[0066]

[0067] Step 5: Load the model parameters obtained in step 4 to obtain the trained network traffic prediction model, input the test set established in step 2 into the model, and obtain the network traffic prediction value.

[0068] Step 6: Evaluate the prediction performance of the trained network traffic prediction model.

[0069] For prediction performance, this paper uses three common performance indicators: mean absolute error (MAE), mean squared error (MSE), and mean absolute percentage error (MAPE) to evaluate the performance of the model. These three indicators can be defined as follows:

[0070]

[0071]

[0072]

[0073] To demonstrate the advantages of the proposed model approach, we compared several commonly used prediction models. The results are shown in Tables 1 and 2. Our proposed SMDCN model outperforms other baseline models in all metrics on the NTBS, WTSW, and AIIA datasets, and ranks second in terms of MSE and MAE on the ISP and EC datasets. This demonstrates excellent prediction performance.

[0074] Table 1: Indicators of the SMDCN model on the NTBS and WTSW datasets

[0075]

[0076] Table 2: Indicators of the SMDCN model on the AIIA, ISP, and EC datasets

[0077]

[0078]

[0079] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A network traffic prediction method based on a multi-scale dilated convolutional neural network, characterized in that: The following steps are involved: Step 1: We obtained five datasets with large noise and outliers, namely NTBS, WTSW, AIIA, ISP, and EC data, and sorted and analyzed the data to make them the input variables. Step 2: Divide the sorted data into a training set and a test set with a ratio of 8:

2. At the same time, perform a sliding average on the training data set, where the sliding window value is set to 3, that is, the average of the three adjacent numbers in the original data is obtained; Step 3: Establish a dilated convolutional neural network prediction model based on random sparse self-attention mechanism; Step 4: Input the training set into the dilated convolutional neural network prediction model with random sparse self-attention mechanism to train the network traffic prediction model. After the training is completed, save the model parameters. Step 5: Load the model parameters to obtain the trained network traffic prediction model, input the test set into the model, and obtain the network traffic prediction value; Step 6: Evaluate the prediction performance of the trained network traffic prediction model; The specific contents of the third step are: S1: Input the input sequence into three branches of the dilated convolutional neural network with convolution kernel sizes of k1, k2 and k3 respectively; S2: Determine the number of convolutional layers N1, N2, and N3 through the layer determination mechanism; S3: A random sparse self-attention mechanism is inserted into the network between every three dilated convolutional layer modules to consider the importance differences of different temporal features, and the results of the three branch networks are connected; S4: Perform linear transformation on the result to obtain the output sequence.

2. The network traffic prediction method using a multi-scale dilated convolutional neural network according to claim 1 is characterized by: The NTBS and WTSW data in the first step contain multiple time series, while the AIIA, ISP, and EC data have only one time series.

3. The network traffic prediction method using a multi-scale dilated convolutional neural network according to claim 1 is characterized by: In the fourth step, the model parameters are set as: The learning rate is set to 0.0001, and the Adam optimizer is used to optimize the network model parameters. The loss function adopts the following loss function. The training cycle is 150 rounds. For the datasets ISP, AIIA, and EC, the length of the input sequence is set to 200. For the datasets NTBS and WTSW, the length of the input sequence is set to 504 and 350 respectively, and the length of the output sequence is set to 168 and 47 respectively; ； ； in are the parameters of the model, is the output sequence length, and N is the number of network layers.

4. The network traffic prediction method using a multi-scale dilated convolutional neural network according to claim 1, characterized in that: In the sixth step, MAE, MSE, and MAPE are used to evaluate the performance of the model. The indicators can be defined as follows: ； ； 。

Images