Large-scale network flow modeling and parallel simulation method

Abstract

The invention discloses a large-scale network traffic modeling and parallel simulation method, which comprises the following steps: firstly, collecting traffic data packet information on a gateway router, extracting header information of the collected traffic data packet, then carrying out traffic identification on the data packet to obtain TCP (Transmission Control Protocol) traffic data, counting traffic characteristics based on network self-similarity and carrying out unsupervised clustering; the method comprises the following steps of: clustering flow with strong correlation into one class, preprocessing each class of clustered flow data, putting each class of flow as training data into a WGAN-GP flow generation model for training through multiple processes to obtain a flow generation model of multiple classes of flow, and enabling all the trained flow generation models to generate flow in parallel through multiple processes. And combining the generated flows to obtain simulation network flows which are distributed complexly and accord with reality. According to the method, large-scale network traffic can be processed, large-scale complex traffic of backbone network traffic can be efficiently processed, and simulated network traffic which is more complex in distribution and more conforms to practical characteristics is generated.

Description

Technical Field

[0001] The invention belongs to the field of computer networks, and in particular relates to a large-scale network traffic modeling and parallel simulation method. Background Art

[0002] With the rapid development of computer networks, the scale of networks has expanded rapidly. The pressure brought by these network traffic to various applications in the network has increased rapidly. For some applications that are not yet mature and have not been tested, traffic is needed to conduct communication tests in advance. Therefore, traffic that can evaluate the design protocol of communication is very important. However, the traffic used for testing is often limited in quantity and type due to certain restrictions. When a large amount of complex traffic is required for testing and evaluation, the traffic obviously cannot meet the requirements, so simulated traffic generation technology came into being. This technology can generate simulated traffic that obeys the distribution of real traffic instead of real traffic, and users can adjust the type, speed and quantity of simulated traffic generation according to their needs, which is suitable for various occasions.

[0003] The original traffic generation model was based on Poisson distribution. This model assumes that the process of packet arrival conforms to the Poisson process. Such a traffic generator model uses probability distribution to model the traffic pattern to calculate the traffic. Obviously, it is impossible to completely simulate the complex packet traffic in the real world. Self-similar traffic modeling appeared later, such as traffic modeling based on fractal Brownian motion (FBM), which is an abstract model of aggregated connectionless traffic based on fractal Brownian motion, and also includes fractal Gaussian noise (FGN) traffic modeling. They are not very consistent with the characteristics of real traffic due to some negative signals. The fractal autoregressive moving average (FARIMA) model can capture both short-range and long-term dependencies, so it can be used to model and predict traffic, but its calculation process is very complicated. There is also a new model for generating self-similar traffic based on a multi-layer perceptron neural network and a random error generator. This model can make the generated traffic have the same self-similarity as the real traffic with the help of the random error generator, but it obviously has the disadvantage that it cannot learn the traffic characteristics more completely when the neural network model is relatively simple.

[0004] Due to the gradual popularization of the Internet and the complexity of network structure, the demand for traffic generation in more and more fields has increased, and the existing traffic generation technology has the following problems: First, with the continuous development of computer technology, network technology and communication technology, more and more traffic of different protocols flows into the network. The current network traffic, especially the traffic in the backbone network, has become more and more complex in distribution, and the existing traffic generation scheme cannot quickly and effectively handle such complex traffic distribution, resulting in its generation efficiency is always relatively low, and cannot meet the requirements of real traffic generation business; second, due to the complex distribution of real traffic, the existing traffic generation method scheme is not reasonable enough, and it is impossible to completely extract and learn the complex distribution of real traffic characteristics. Usually, it can only generate traffic with relatively simple distribution, and cannot generate traffic data that can simulate the complex distribution in the real world. Therefore, the traffic generated by the existing traffic generation method is often not reasonable and real enough. Therefore, it is extremely important to generate a solution that can generate simulated traffic with complex distribution, many types and more in line with the real distribution. Summary of the invention

[0005] To solve the above technical problems, the present invention provides a large-scale network traffic modeling and parallel simulation method, which can process large-scale complex traffic such as backbone network traffic with higher efficiency, and can generate simulated network traffic with more complex distribution and more in line with realistic characteristics.

[0006] The technical solution adopted by the present invention is: a large-scale network traffic modeling and parallel simulation method, the specific steps are as follows:

[0007] S1. Design a packet capture program based on Libcap, deploy the packet capture program on the gateway router to collect information about traffic data packets, and extract the header information of the collected traffic data packets;

[0008] S2, based on the header information obtained in step S1, perform flow identification on the data packet to obtain TCP flow data;

[0009] S3, based on the TCP flow data obtained in step S2, selecting and extracting network-based self-similarity statistical flow features;

[0010] S4, based on the flow features counted in step S3, unsupervised clustering is performed based on MiniBatch KMeans to cluster network flows with strong feature correlation into one category;

[0011] Among them, the cluster number parameter of MiniBatch KMeans is set to 900.

[0012] S5, preprocessing each type of traffic data after clustering in step S4, and then building a traffic generation model based on WGAN-GP, putting each type of traffic as training data into the WGAN-GP traffic generation model through multiple processes for parallel training, and obtaining traffic generation models for multiple types of traffic;

[0013] S6. Based on step S5, all trained WGAN-GP traffic generation models are again allowed to generate traffic in parallel through multiple processes, and the traffic generated by each is merged together, and finally a simulated network traffic with complex distribution and in line with reality is obtained.

[0014] Furthermore, the step S1 is specifically as follows:

[0015] Firstly, a packet capture program is designed based on Libcap, and the functions in Libcap are used to capture only the header information of the data packet as the basic information for collecting traffic, and the packet capture program only collects the information of the header of the data packet.

[0016] Then cross-compile the packet capture program and deploy it on the gateway router, select the specified network card for monitoring, and continuously capture traffic data packets.

[0017] Among them, the gateway router is selected as a soft router and the operating system is the OpenWrt system.

[0018] The packet capture program takes the obtained data packet data in binary form, so a binary file reading program is designed to read the binary file of the collected data packet data into a txt file.

[0019] The txt file records the detailed header information of each data packet collected.

[0020] Furthermore, the step S2 is specifically as follows:

[0021] Based on the traffic data packets collected in step S1, firstly, data packets belonging to the TCP protocol and the UDP protocol are separated according to the protocol field, the data packets belonging to the UDP protocol are discarded, and the data packets belonging to the TCP protocol are subjected to flow identification.

[0022] Among them, flow identification is to analyze each packet and identify the flow to which the packet belongs based on the five fields of IP address, destination IP address, source port, destination port, and protocol in the header information of the data packet.

[0023] Then, packets with the same (IP address, destination IP address, source port, destination port) four-tuple or the same (destination IP address, IP address, destination port, source port) four-tuple are assigned to the same flow, and the obtained flow is stored using a data structure dictionary.

[0024] The key of the dictionary is a four-tuple of (IP address, destination IP address, source port, destination port), and the value is a custom class used to represent the flow. This custom class uses a list to store all the data packets in the flow, and uses other member variables and member functions to count other features of the flow.

[0025] Finally, after the self-defined class statistics are processed, flow filtering is performed to remove TCP flows that have only one data packet, no syn message, handshake failure, or no normal end due to special circumstances, leaving only normal and healthy TCP flows.

[0026] Furthermore, the step S3 is specifically as follows:

[0027] The original traffic data is parsed, and 52 packet-level features are finally selected as the final features of a TCP flow, including: number of uplink / downlink packets, total number of packets, total packet size, uplink / downlink packet size, Q1 quantile of uplink / downlink packet arrival time interval, Q3 quantile of uplink / downlink packet arrival time interval, mean uplink / downlink packet arrival time interval, maximum uplink / downlink packet size, minimum uplink / downlink packet size, maximum uplink / downlink packet interval, minimum uplink / downlink packet interval, median uplink / downlink packet size, mean uplink / downlink packet size, mode uplink / downlink packet size, median uplink / downlink packet interval, mean uplink / downlink packet interval, mode uplink / downlink packet interval, range of uplink / downlink packet size, variance of uplink / downlink packet size, range of uplink / downlink packet interval, variance of uplink / downlink packet interval, skewness of uplink / downlink packet size, kurtosis of uplink / downlink packet size, skewness of uplink / downlink packet interval, kurtosis of uplink / downlink packet interval, IP address, destination IP address, source port, destination port.

[0028] Furthermore, the step S5 is specifically as follows:

[0029] The WGAN-GP traffic generation model includes: a generator network module and a discriminator network module.

[0030] Among them, the input vector dimension of the generator network module is 32-dimensional, and the output vector dimension is 2-dimensional, including 4 linear layers and 3 RELU layers; the input vector dimension of the discriminator network module is 2-dimensional, and the output vector dimension is 1-dimensional, including 4 linear layers and 3 RELU layers.

[0031] The TCP flow data after clustering in step S4 is split and calculated in their respective classes, the flow is split back into data packets and the packet size characteristics and packet interval time characteristics of each data packet in each class are obtained, each packet is modeled using the packet size characteristics and packet interval time characteristics, and then the above two characteristics of the data packets in each class are standardized.

[0032] Among them, the standardized data packets will be used for WGAN-GP traffic generation model training, and one WGAN-GP traffic generation model will be trained for each category.

[0033] The standardized data packet data is batch processed according to a batch size of 64. The batch processed data is directly used as the training data of the WGAN-GP traffic generation model. Finally, multiple types of data are trained in parallel through multiple processes.

[0034] During the training process, the WGAN-GP traffic generation model generates a standard Gaussian random variable of length 32 as the input of the generator, so that it generates a vector of length 2 based on the input. The vector represents a generated data packet, which is a false sample. The generated false sample and the real data packet are then input into the discriminator together to distinguish them.

[0035] Among them, the generator loss function expression is as follows:

[0036] Loss G = -D(G(z))

[0037] Among them, G(z) represents the output of the generator, that is, the fake sample generated by the generator, and D(G(z)) represents the output of the discriminator obtained by taking the output of the generator as the input of the discriminator.

[0038] The loss function expression of the discriminator is as follows:

[0039] Loss D =D(G(z))-D(x)+weight*GP

[0040] Among them, z represents the input 32-dimensional standard Gaussian random variable, x represents the actual data packet data, GP represents the gradient penalty term, and weight represents the weight of the gradient penalty term set according to the actual situation.

[0041] During the training of the WGAN-GP traffic generation model, the training strategies adopted are as follows:

[0042] First, the generator parameters are fixed, and the weight of the gradient penalty term is set to 0.001. The parameters of the discriminator are updated using a stochastic gradient descent optimizer with a learning rate of 0.0001. And every 5 times the discriminator is updated, the parameters of the discriminator are fixed and the parameters of the generator are updated once using a stochastic gradient descent optimizer with a learning rate of 0.0001. Finally, the number of discriminator training rounds is set to 12,000 rounds, and the generator is trained for 2,400 rounds.

[0043] After the WGAN-GP traffic generation model is trained, a scatter plot and probability density comparison chart of the generated traffic and the real traffic are drawn to judge the generation effect of the model.

[0044] Beneficial effects of the present invention: The method of the present invention first collects traffic data packet information on the gateway router, and extracts the header information of the collected traffic data packets, then performs flow identification on the data packets to obtain TCP flow data, statistically analyzes flow characteristics based on network self-similarity and performs unsupervised clustering, clusters highly correlated flows into one category, pre-processes each category of traffic data after clustering, and then uses multiple processes to put each category of traffic as training data into the WGAN-GP traffic generation model for training, obtains traffic generation models for multiple categories of traffic, and uses multiple processes to allow all trained traffic generation models to generate traffic in parallel, merges the traffic generated by each, and obtains simulated network traffic with complex distribution and in line with reality. The method of the present invention can process large-scale network traffic, efficiently process large-scale complex traffic of backbone network traffic, and generate simulated network traffic with more complex distribution and more in line with reality.

[0045] The method of the present invention clusters the collected large-scale network traffic with complex characteristics using an unsupervised learning method, identifies self-similar traffic, reduces the complexity of traffic characteristics, and separately models each type of traffic with relatively simple characteristics, thereby improving the accuracy of traffic modeling and subsequent traffic generation. The method of the present invention uses the WGAN-GP traffic generation model to train and generate each type of traffic with relatively simple characteristics, which has a better and more stable generation effect than the existing model. The parallel generation of multiple models through multiple processes can not only improve the efficiency of traffic generation, but also control the size of the generated traffic and the categories included by controlling the number of parallelism, thereby realizing flexible generation of traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The present invention is a flowchart of a large-scale network traffic modeling and parallel simulation method.

[0047] Figure 2 Schematic diagram of the WGAN-GP generator and discriminator network structure in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The method of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0049] like Figure 1 As shown, a flow chart of a large-scale network traffic modeling and parallel simulation method of the present invention, the specific steps are as follows:

[0050] S1. Design a packet capture program based on Libcap, deploy the packet capture program on the gateway router to collect information about traffic data packets, and extract the header information of the collected traffic data packets;

[0051] S2, based on the header information obtained in step S1, perform flow identification on the data packet to obtain TCP flow data;

[0052] S3, based on the TCP flow data obtained in step S2, selecting and extracting network-based self-similarity statistical flow features;

[0053] S4, based on the flow features counted in step S3, unsupervised clustering is performed based on MiniBatch KMeans to cluster network flows with strong feature correlation into one category;

[0054] Among them, the cluster number parameter of MiniBatch KMeans is set to 900.

[0055] S5, preprocessing each type of traffic data after clustering in step S4, and then building a traffic generation model based on WGAN-GP, putting each type of traffic as training data into the WGAN-GP traffic generation model through multiple processes for parallel training, and obtaining traffic generation models for multiple types of traffic;

[0056] S6. Based on step S5, all trained WGAN-GP traffic generation models are again allowed to generate traffic in parallel through multiple processes, and the traffic generated by each is merged together, and finally a simulated network traffic with complex distribution and in line with reality is obtained.

[0057] In this embodiment, the step S1 is specifically as follows:

[0058] Currently, the most common packet capture tool on Linux systems is TCPDUMP. However, since the amount of data to be collected in this embodiment is relatively large, using TCPDUMP for traffic collection requires a large physical storage space, and the processing of the collected traffic will also be relatively cumbersome. In order to facilitate storage and processing, this embodiment designs a packet capture program based on Libcap (Libcap refers to a network data packet capture function library under the Linux platform, which is an API interface for user-level packet capture independent of the system, and provides a portable framework for underlying network monitoring). The functions in Libcap are used to capture only the header information of the data packet as the basic information for collecting traffic (reducing the amount of data collected, while recording the important feature content of each data packet). The data load part of the data packet usually involves the privacy of network users. The packet capture program only collects the information of the data packet header, which plays a certain role in protecting the privacy information of user data.

[0059] Then cross-compile the packet capture program and deploy it on the gateway router, select the specified network card for monitoring, and continuously capture traffic data packets.

[0060] In order to deploy the packet capture program on the gateway router, the gateway router is selected as a soft router, and the operating system is the OpenWrt system (embedded Linux system).

[0061] The packet capture program takes the obtained data packet data in binary form, so a binary file reading program is designed to read the binary file of the collected data packet data into a txt file.

[0062] The txt file records the detailed header information of each data packet collected, and the header information of the data packet is obtained by reading the txt file.

[0063] In this embodiment, step S2 is specifically as follows:

[0064] Based on the traffic data packets collected in step S1, firstly, data packets belonging to the TCP protocol and the UDP protocol are separated according to the protocol field, and the data packets belonging to the UDP protocol are discarded (corresponding to the connection-oriented characteristics of the TCP protocol and the connectionless characteristics of the UDP protocol in real network traffic), and the data packets belonging to the TCP protocol are subjected to flow identification.

[0065] Among them, flow identification is to analyze each packet and identify the flow to which the packet belongs based on the five fields of IP address, destination IP address, source port, destination port, and protocol in the header information of the data packet.

[0066] Then, packets with the same (IP address, destination IP address, source port, destination port) four-tuple or the same (destination IP address, IP address, destination port, source port) four-tuple are assigned to the same flow, and the obtained flow is stored using a data structure dictionary.

[0067] The key of the dictionary is a four-tuple of (IP address, destination IP address, source port, destination port), and the value is a custom class used to represent the flow. This custom class uses a list to store all the data packets in the flow, and uses other member variables and member functions to count other features of the flow.

[0068] Finally, after the self-defined class statistics are processed, flow filtering is performed to remove TCP flows that have only one data packet, no syn message, handshake failure, or no normal end due to special circumstances, leaving only normal and healthy TCP flows.

[0069] In this embodiment, step S3 is specifically as follows:

[0070] Since the collected traffic features are distributed in a complex manner, if they are directly used for the training of the traffic generation model, it will lead to problems such as poor traffic generation effect and low generation efficiency. The information network has self-similarity, which means that certain features in a complex network show similar properties at different scales. This means that whether a part of a complex network is enlarged or reduced, similar patterns and features can be presented at different scales. Therefore, this embodiment will extract the pre-designed features in each TCP stream, use these features to perform unsupervised clustering processing on the collected traffic, and then separately model each type of traffic data with relatively simple feature distribution, thereby improving the accuracy of traffic modeling and the effect of subsequent traffic generation. In real traffic, for different hosts accessing the same web page resource, or downloading the same content on a web page, their total number of data packets, data packet size, ratio of upstream and downstream data packets and other information are highly similar, and these features can effectively reflect the traffic behavior of each TCP stream.

[0071] By parsing the original traffic data, 52 packet-level features are finally selected as the final features of a TCP flow, including: the number of uplink / downlink packets, the total number of packets, the total packet size, the uplink / downlink packet size, the Q1 quantile of the uplink / downlink packet arrival time interval, the Q3 quantile of the uplink / downlink packet arrival time interval, the mean of the uplink / downlink packet arrival time interval, the maximum of the uplink / downlink packet size, the minimum of the uplink / downlink packet size, the maximum of the uplink / downlink packet interval, the minimum of the uplink / downlink packet interval, the median of the uplink / downlink packet size, the average of the uplink / downlink packet size, the mode of the uplink / downlink packet size, the median of the uplink / downlink packet interval, the average of the uplink / downlink packet interval, the mode of the uplink / downlink packet interval, the range of the uplink / downlink packet size, the variance of the uplink / downlink packet size, the range of the uplink / downlink packet interval, the variance of the uplink / downlink packet interval, the skewness of the uplink / downlink packet size, the kurtosis of the uplink / downlink packet size, the skewness of the uplink / downlink packet interval, the kurtosis of the uplink / downlink packet interval, the IP address, the destination IP address, the source port, and the destination port.

[0072] In this embodiment, step S4 is specifically as follows:

[0073] After extracting the corresponding features of the network traffic, the network traffic with greater feature correlation can be clustered into one category through a clustering algorithm.

[0074] There are many clustering algorithms. Among them are partition-based clustering algorithms, such as KMedoids and KMeans; density-based clustering algorithms, such as DBSCAN and OPTICS; hierarchical clustering algorithms, such as BIRCH and CURE, etc. In this embodiment, the traffic data is first counted in advance through the port, IP and application layer protocol fields to obtain the categories of the collected traffic. Finally, the traffic can be roughly divided into https, http, server traffic and other traffic. Then, each clustering method is used to cluster the traffic data. After counting which type of traffic is the largest in each type of traffic after clustering, this type of traffic is defined as this type of traffic. Finally, the precision, recall and f1 scores of these four types of traffic are calculated, and finally the clustering method is selected based on the situation of these three indicators. Finally, after multiple rounds of testing and debugging, we eliminated methods such as OPTICS and DBSCAN that have too slow clustering speed and too high memory requirements, and methods such as Clique and WaveCluster that have too poor clustering effects. Finally, we found that the MiniBatch KMeans clustering method has a clustering time of less than one minute each time, which is very fast, and the precision, recall rate and f1 score will increase with the increase of the value of the cluster number parameter. When the cluster number is set to 900, the precision, recall rate and f1 score all reach above 0.85. Therefore, this embodiment finally uses MiniBatch KMeans as the clustering algorithm for traffic classification.

[0075] The existing KMeans algorithm is popular because of its simplicity and efficiency. The basic idea of ​​the KMeans algorithm is to divide the data into K different clusters in an iterative way, and minimize the sum of the distances between each data point and the centroid (or center point, mean point) of the cluster to which it belongs. Specifically, the execution process of the KMeans algorithm usually includes the following steps: first, randomly select K data points as the initial cluster centroids; then, assign each data point to the nearest cluster according to its distance from the centroid of each cluster; then, recalculate the centroid of each cluster, that is, take the average of all data points in the cluster as the new centroid; repeat the above assignment and update steps until a certain termination condition is met (such as the cluster centroid no longer changes significantly or reaches a preset number of iterations). The MiniBatch KMeans algorithm is a variant of the KMeans algorithm that uses mini-batches to reduce computing time, while these multiple batches still try to optimize the same objective function. Mini-batches are subsets of the input data, randomly sampled in each training iteration. These mini-batches greatly reduce the amount of computation required to converge to a local solution. Unlike other algorithms that reduce the convergence time of KMeans, mini-batch KMeans will inevitably lead to a decrease in accuracy due to sampling, but the results produced are usually only slightly worse than the standard algorithm.

[0076] The MiniBatch KMeans algorithm can effectively extract signal traffic with strong feature correlation when the sample size is very large. The overall method flow is: collect network traffic, obtain TCP / IP header field information, extract the above 52 flow features of each flow, and perform MiniBatch KMeans clustering on network traffic based on these features, where the cluster number parameter of MiniBatch KMeans is set to 900, and finally model different types of traffic after clustering separately and use them as subsequent data for traffic generation.

[0077] In this embodiment, step S5 is specifically as follows:

[0078] The existing traffic generator model cannot fully simulate the complex traffic data in the real world. For example, the traffic model based on Poisson distribution only uses Poisson distribution to fit the generation process of real traffic. The self-similar model also has similar problems. With the development of artificial intelligence and neural networks, the powerful fitting ability of the neural network model makes it the choice of traffic generation model, and the neural network generation model is the first choice for the task of generating traffic. WGAN-GP is a very famous model in the neural network generation model. This embodiment uses a parallel traffic generation method based on the WGAN-GP model, which will have a more stable generation effect than using the existing traffic generation model. Using multiple processes to generate traffic in parallel can improve the efficiency of traffic generation and make the final generated traffic distribution closer to the distribution of real traffic.

[0079] like Figure 2 As shown, the WGAN-GP traffic generation model includes: a generator network module and a discriminator network module.

[0080] Among them, the input vector dimension of the generator network module is 32-dimensional, and the output vector dimension is 2-dimensional, including 4 linear layers and 3 RELU layers; the input vector dimension of the discriminator network module is 2-dimensional, and the output vector dimension is 1-dimensional, including 4 linear layers and 3 RELU layers.

[0081] This embodiment obtains a generative model for realizing traffic generation by training the WGAN-GP model, and obtains a traffic generator with good effect by continuously adjusting hyperparameters and model structure details and in the mutual confrontation process between the generator and the discriminator.

[0082] The TCP flow data after clustering in step S4 is split and calculated in their respective classes, the flow is split back into data packets and the packet size characteristics and packet interval time characteristics of each data packet in each class are obtained, and each packet is modeled using the packet size characteristics and packet interval time characteristics (different categories of traffic are modeled separately after clustering), and then the above two characteristics of the data packets in each class are standardized.

[0083] Among them, the standardized data packets will be used for WGAN-GP traffic generation model training, and one WGAN-GP traffic generation model will be trained for each category.

[0084] The standardized data packet data is batch processed according to a batch size of 64. The batch processed data is directly used as the training data of the WGAN-GP traffic generation model. Finally, multiple types of data are trained in parallel through multiple processes.

[0085] During the training process, the WGAN-GP traffic generation model generates a standard Gaussian random variable of length 32 as the input of the generator, so that it generates a vector of length 2 based on the input. The vector represents a generated data packet, which is a false sample. The generated false sample and the real data packet are then input into the discriminator together to distinguish them.

[0086] Among them, the generator loss function expression is as follows:

[0087] Loss G = -D(G(z))

[0088] Among them, G(z) represents the output of the generator, that is, the fake sample generated by the generator, and D(G(z)) represents the output of the discriminator obtained by taking the output of the generator as the input of the discriminator.

[0089] The loss function expression of the discriminator is as follows:

[0090] Loss D =D(G(z))-D(x)+weight*GP

[0091] Among them, z represents the input 32-dimensional standard Gaussian random variable, x represents the actual data packet data, GP represents the gradient penalty term, and weight represents the weight of the gradient penalty term set according to the actual situation.

[0092] According to the loss function, the adversarial process between the generator and the discriminator is actually to distinguish between real samples and generated samples by giving higher scores to real samples and lower scores to generated samples, and to limit the gradient of the discriminator update through the gradient penalty term to ensure the Lipschitz continuity condition. The generator hopes that the samples it generates can score higher in the discriminator to deceive the discriminator, hoping that the discriminator will regard the fake samples it generates as real samples.

[0093] During the training of the WGAN-GP traffic generation model, the training strategies adopted are as follows:

[0094] First, the generator parameters are fixed, and the weight of the gradient penalty term is set to 0.001. The parameters of the discriminator are updated using a stochastic gradient descent optimizer with a learning rate of 0.0001. And every 5 times the discriminator is updated, the parameters of the discriminator are fixed and the parameters of the generator are updated once using a stochastic gradient descent optimizer with a learning rate of 0.0001. Finally, the number of discriminator training rounds is set to 12,000 rounds, and the generator is trained for 2,400 rounds, and finally a WGAN-GP model with good generation effect is obtained.

[0095] After the WGAN-GP traffic generation model is trained, a scatter plot and probability density comparison chart of the generated traffic and the real traffic are drawn to judge the generation effect of the model.

[0096] In this embodiment, step S6 is specifically as follows:

[0097] After obtaining the traffic generation model, multiple processes are used to allow the various WGAN-GP traffic generation models trained using different types of traffic to generate traffic in parallel and merge the output. This traffic generation method not only has extremely high generation efficiency and can generate large-scale traffic in a short time, but also can adjust the size of the generated traffic and the traffic categories contained in the generated traffic by adjusting the number of processes. This not only does not affect the effect of generating traffic, but also can flexibly and effectively control the output of traffic according to one's own needs.

[0098] In summary, the method of the present invention clusters the collected large-scale network traffic with complex characteristics using an unsupervised learning method, identifies self-similar traffic, reduces the complexity of traffic characteristics, and separately models each type of traffic with relatively simple characteristics, thereby improving the accuracy of traffic modeling and subsequent traffic generation. The method of the present invention uses the WGAN-GP traffic generation model to train and generate each type of traffic with relatively simple characteristics, which has a better and more stable generation effect than the existing model. The parallel generation of multiple models through multiple processes can not only improve the efficiency of traffic generation, but also control the size and categories of generated traffic by controlling the number of parallel processes, thereby achieving flexible generation of traffic.

[0099] The principle of the present invention should be understood that the protection scope of the present invention is not limited to such special statements and embodiments. Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand that those skilled in the art can make various other specific variations and combinations based on the technical revelations disclosed by the present invention without departing from the essence of the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A large-scale network traffic modeling and parallel simulation method, the specific steps are as follows: S1. Design a packet capture program based on Libcap, deploy the packet capture program on the gateway router to collect information about traffic data packets, and extract the header information of the collected traffic data packets; S2, based on the header information obtained in step S1, perform flow identification on the data packet to obtain TCP flow data; S3, based on the TCP flow data obtained in step S2, selecting and extracting network-based self-similarity statistical flow features; S4, based on the flow features counted in step S3, unsupervised clustering is performed based on MiniBatch KMeans to cluster network flows with strong feature correlation into one category; Among them, the cluster number parameter of MiniBatch KMeans is set to 900; S5, preprocessing each type of traffic data after clustering in step S4, and then building a traffic generation model based on WGAN-GP, putting each type of traffic as training data into the WGAN-GP traffic generation model through multiple processes for parallel training, and obtaining traffic generation models for multiple types of traffic; S6. Based on step S5, all trained WGAN-GP traffic generation models are again allowed to generate traffic in parallel through multiple processes, and the traffic generated by each is merged together, and finally a simulated network traffic with complex distribution and in line with reality is obtained.

2. A large-scale network traffic modeling and parallel simulation method according to claim 1, It is characterized in that The step S1 is specifically as follows: Firstly, a packet capture program is designed based on Libcap, and the functions in Libcap are used to capture only the header information of the data packet as the basic information of the collected traffic, and the packet capture program only collects the information of the header of the data packet; Then, the packet capture program is cross-compiled and deployed on the gateway router, and the specified network card is selected for monitoring to continuously capture traffic data packets; Among them, the gateway router is selected as a soft router, and the operating system is the OpenWrt system; The packet capture program takes the obtained data packet data in binary form, so a binary file reading program is designed to read the binary file of the collected data packet data into a txt file; The txt file records the detailed header information of each data packet collected.

3. A large-scale network traffic modeling and parallel simulation method according to claim 1, It is characterized in that The step S2 is specifically as follows: Based on the traffic data packets collected in step S1, firstly, data packets belonging to the TCP protocol and the UDP protocol are separated according to the protocol field, the data packets belonging to the UDP protocol are discarded, and the data packets belonging to the TCP protocol are subjected to flow identification; Among them, flow identification is to analyze each packet and identify the flow to which the packet belongs based on the five fields of IP address, destination IP address, source port, destination port, and protocol in the header information of the data packet; Then, the packets with the same (IP address, destination IP address, source port, destination port) or (destination IP address, IP address, destination port, source port) four-tuple are classified into the same flow, and the obtained flow is stored in a data structure dictionary; The key of the dictionary is a four-tuple of (IP address, destination IP address, source port, destination port), and the value is a custom class used to represent the flow. The custom class uses a list to store all the data packets in the flow, and uses other member variables and member functions to count other features of the flow. Finally, after the self-defined class statistics are processed, flow filtering is performed to remove TCP flows that have only one data packet, no syn message, handshake failure, or no normal end due to special circumstances, leaving only normal and healthy TCP flows.

4. A large-scale network traffic modeling and parallel simulation method according to claim 1, It is characterized in that The step S3 is specifically as follows: The original traffic data is parsed, and 52 packet-level features are finally selected as the final features of a TCP flow, including: number of uplink / downlink packets, total number of packets, total packet size, uplink / downlink packet size, Q1 quantile of uplink / downlink packet arrival time interval, Q3 quantile of uplink / downlink packet arrival time interval, mean uplink / downlink packet arrival time interval, maximum uplink / downlink packet size, minimum uplink / downlink packet size, maximum uplink / downlink packet interval, minimum uplink / downlink packet interval, median uplink / downlink packet size, mean uplink / downlink packet size, mode uplink / downlink packet size, median uplink / downlink packet interval, mean uplink / downlink packet interval, mode uplink / downlink packet interval, range of uplink / downlink packet size, variance of uplink / downlink packet size, range of uplink / downlink packet interval, variance of uplink / downlink packet interval, skewness of uplink / downlink packet size, kurtosis of uplink / downlink packet size, skewness of uplink / downlink packet interval, kurtosis of uplink / downlink packet interval, IP address, destination IP address, source port, destination port.

5. A large-scale network traffic modeling and parallel simulation method according to claim 1, It is characterized in that The step S5 is specifically as follows: The WGAN-GP traffic generation model includes: a generator network module and a discriminator network module; Among them, the input vector dimension of the generator network module is 32-dimensional, the output vector dimension is 2-dimensional, including 4 linear layers and 3 RELU layers; the input vector dimension of the discriminator network module is 2-dimensional, the output vector dimension is 1-dimensional, including 4 linear layers and 3 RELU layers; The TCP flow data after clustering in step S4 is split and calculated in each class respectively, the flow is split back into data packets and the packet size feature and packet interval time feature of each data packet in each class are obtained, each packet is modeled using the packet size feature and packet interval time feature, and then the above two features of the data packets in each class are standardized; The standardized data packets will be used for WGAN-GP traffic generation model training. One WGAN-GP traffic generation model will be trained for each category. The standardized data packets are processed in batches with a batch size of 64. The processed data are directly used as the training data for the WGAN-GP traffic generation model. Finally, multiple processes are used to train multiple types of data in parallel. During the training process, the WGAN-GP traffic generation model generates a 32-dimensional standard Gaussian random variable as the input of the generator, which generates a 2-dimensional vector based on the input. The vector represents a generated data packet, which is a fake sample. The generated fake sample and the real data packet are then input into the discriminator together to distinguish them. Among them, the generator loss function expression is as follows: Loss G =-D(G(z)) Among them, G(z) represents the output of the generator, that is, the fake sample generated by the generator, and D(G(z)) represents the output of the discriminator obtained by taking the output of the generator as the input of the discriminator; The loss function expression of the discriminator is as follows: Loss D =D(G(z))-D(x)+weight*GP Where z represents the input 32-dimensional standard Gaussian random variable, x represents the real data packet data, GP represents the gradient penalty term, and weight represents the weight of the gradient penalty term set according to the actual situation; During the training of the WGAN-GP traffic generation model, the training strategies adopted are as follows: First, the generator parameters are fixed, and the weight of the gradient penalty term is set to 0.

001. The parameters of the discriminator are updated using a stochastic gradient descent optimizer with a learning rate of 0.0001. After every 5 updates of the discriminator, the parameters of the discriminator are fixed and the parameters of the generator are updated once using a stochastic gradient descent optimizer with a learning rate of 0.0001. Finally, the number of discriminator training rounds is set to 12,000 rounds, and the generator is trained for 2,400 rounds. After the WGAN-GP traffic generation model is trained, a scatter plot and probability density comparison chart of the generated traffic and the real traffic are drawn to judge the generation effect of the model.

Images