A large-scale batch data collection method, system, device and medium for network simulator

Abstract

A large-scale batch data collection method, system, device and medium for network simulator, the method comprising: constructing a network configuration module, constructing a traffic generation module, that is, a traffic component library, generating a corresponding scene description file, constructing a scene description component module, that is, a scene description component library, building a custom network simulation module and a data statistics module, simulating the scene description file on the built custom network simulation module, counting the indicators through the data statistics module, generating a data indicator statistics file, corresponding integration of the scene description file and the generated data indicator statistics file, obtaining a large-scale data sample set for different network scenarios; the system, device and medium are used to realize a large-scale batch data collection method for network simulator; the application has the characteristics of flexible use, fast generation and real reflection of network operation mechanism.

Description

Technical Field

[0001] This invention relates to the field of data acquisition technology, and specifically to a method, system, device, and medium for large-scale batch data acquisition for network simulators. Background Technology

[0002] In recent years, with the rapid development of communication technology, network scale has continued to expand, and network traffic has also shown a surge. These changes have made the network environment more complex and increased the uncertainty between networks. Given the rapid development of networks, establishing accurate network evaluation models for real networks is of great significance. Currently, the most widely used network simulators are packet-level event-based network simulators. Various network simulators (such as NS, NS2, NS3, etc.) and network simulators (such as Mininet, OMNet++, OPNET, etc.) developed by the military, enterprises, and universities have been applied to varying degrees in network simulation performance evaluation.

[0003] The NS network simulator originated from the US military's Real Network Simulator project in 1989 and is one of the earliest network simulators. Currently, commonly used versions of the NS network simulator have been updated to NS-2 and NS-3, which are free to use in research, development, and education, and have extensive wiki documentation. Networkx, developed by NIST (National Institute of Standards and Technology), also includes a toolkit for a graphical front-end for network modeling; however, Networkx is not specifically designed for computer network modeling but rather handles various graphical design and testing tasks.

[0004] In academia, J-SIM, a Java-based network simulation software developed by the Ohio State University research team, offers significantly better stability than NS2 due to the absence of memory leaks and misreading issues in the Java language. OPNET Modeler, one of the four major network simulation product series from OPNET Technology, is primarily developed for network design professionals needing to simulate large and complex networks. J-SIM lacks guidance documentation for large-scale simulations, making it difficult for users to obtain adequate support for complex network development. JiST / SWANS's biggest advantage is its high efficiency, outperforming other simulation platforms in both speed and memory consumption; however, it lacks visualization support, making result analysis inconvenient and requiring a significant learning curve. QualNet, a network simulation software developed by MIT in 1986, employs parallel design and can simulate large wireless networks with thousands of nodes, hundreds of times faster than real software, supporting real-time communication with people, devices, and software in real networks. However, it is extremely expensive and not open-source, currently primarily used by technical personnel in large commercial companies and research institutes.

[0005] Discrete event-based network simulators can provide fine-grained performance simulation results and have a certain degree of flexibility. However, for the large-scale data samples required in the field of graph neural networks, traditional network simulators cannot achieve large-scale simulation, thus making it impossible to quickly generate data samples for large-scale random network scenarios.

[0006] To address the challenge of processing non-Euclidean domain data representing complex relationships between objects, graph neural networks (GNNs) have been proposed for handling graph-structured data and have been widely applied in fields such as social networks, knowledge graphs, and transportation systems. GNNs can more accurately and flexibly model the dependencies between nodes in a graph. In computer networks, various entities, such as forwarding nodes and links, have dependencies. Applying GNNs to computer network modeling allows for accurate modeling of these dependencies. However, like other deep learning models, training GNN models requires large-scale datasets to improve their performance and generalization ability. In practical applications, the size of the dataset often determines the model's performance and effectiveness. However, most currently available data acquisition methods only retain relevant characteristics of various traffic types in the network (latency, jitter, etc.) during the acquisition process, lacking corresponding network scenario and configuration information, which is detrimental to training GNNs to network scenarios.

[0007] Existing data acquisition methods often only retain network characteristics (latency, jitter, etc.) during the data collection process, lacking corresponding network scenario and configuration information. However, AI-based network modeling and performance evaluation models require modeling and simulating various network functions, such as basic network functions and traffic transmission functions, to achieve a realistic simulation of the network scenario. Therefore, traditional data acquisition methods cannot meet the needs of network modeling.

[0008] Most existing network simulators require customized design of modules based on different scenario types, such as routing strategies and scheduling strategies. Therefore, they have certain limitations when simulating multiple scenarios together. They cannot quickly realize corresponding network simulations according to different network scenarios, nor can they achieve large-scale batch data sampling.

[0009] The paper "RouteNet: Leveraging Graph Neural Networks for Network Modeling and Optimization in SDN" provides a method for data acquisition using the OMNet++ simulator. This method involves the configuration and simulation of different input topologies, routing configurations, and traffic patterns. Each data sample includes accurate measurements of relevant end-to-end key performance indicators generated by the simulation, and contains statistics on packet-level latency, jitter, and loss for each source / destination pair. However, this method ignores the differences between traffic types during acquisition and cannot simulate the differentiated traffic in real networks. Therefore, its use is quite limited and it is not suitable for network scenarios where diverse service traffic coexists.

[0010] The announcement [CN114124470A] discloses a network traffic metadata collection technology algorithm, including the following steps: S1, constructing a physical architecture for network traffic collection; S2, dividing system functions and defining classes; S3, monitoring and collecting network traffic information; S4, data fragmentation; S5, data reassembly, reconnecting the data fragments divided during transmission in their original order and restoring them to their original form; S6, data packet transmission, filtering out some data packets before sending them to the application by setting a network data packet filter, while simultaneously collecting information during network communication. However, this collection method only focuses on business traffic information in the network scenario, ignoring basic network configurations such as routing policies and scheduling policies. Therefore, it cannot be used to collect complete network data and is not suitable for AI-based network simulation and performance modeling. Summary of the Invention

[0011] To overcome the shortcomings of the prior art, the present invention aims to provide a method, system, device, and medium for large-scale batch data acquisition for network simulators. By abstracting and simulating the components in a network scenario, the corresponding network scenario is decomposed into different functional modules. Based on the network simulator, the function of customizing the network according to different scenarios is realized. According to different configuration information, large-scale data samples are quickly generated to provide data support for model training in the field of graph neural networks, meet the needs of network modeling, realize large-scale quantitative data acquisition, and collect complete network data. Therefore, it has the characteristics of flexible use, fast generation speed, and real reflection of the network operation mechanism.

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0013] A method for large-scale batch data acquisition for network simulators includes the following steps:

[0014] Step 1: Based on relevant network knowledge in the network scenario, construct a network configuration module, including a basic topology component library and a routing algorithm component library;

[0015] Step 2: Based on the traffic characteristics in the network scenario, select different traffic data samples from the public traffic dataset for processing to build a traffic generation module, i.e., a traffic component library;

[0016] Step 3: Based on the characteristics of different network scenarios, combine the components in the network configuration module built in Step 1, select the basic topology components and routing algorithm components that conform to the characteristics of the network scenario to form an abstract network description, select traffic samples from the traffic generation module built in Step 2 as traffic inputs in the abstract network to form a traffic configuration description, and combine the abstract network description and the traffic configuration description to generate the corresponding scenario description file.

[0017] Step 4: Repeat step 3 to quickly generate large-scale scene description files and build scene description modules, i.e., scene description component libraries;

[0018] Step 5: In a real network or network simulator, build a custom network simulation module and data statistics module;

[0019] Step 6: On the custom network simulation module built in step 5, simulate the scenario description file in step 4, and use the data statistics module to collect statistics on the indicators and generate a data indicator statistics file.

[0020] Step 7: Integrate the scenario description file from Step 4 with the data indicator statistics file generated in Step 6 to obtain a large-scale data sample set for different network scenarios.

[0021] The network configuration module in step 1 is a basic topology component library and routing algorithm component library required in a specific network scenario. It is responsible for setting and planning the basic network structure and forwarding functions in the network scenario to ensure the realization of basic network functions.

[0022] The basic topology component library mainly includes node information and link information, and its basic characteristics are described below:

[0023] Node information: includes node location, number of ports, scheduling policy node configuration information at the ports;

[0024] Link information includes link source and destination, link packet loss rate, link bandwidth, link latency, and other link characteristics.

[0025] Routing algorithm component library: Includes the planning of routing and forwarding logic in the entire network topology, configured in the form of static routing tables, and dynamically generated according to different routing algorithms.

[0026] The traffic generation module in step 2 performs feature cleaning and generalization based on various publicly available real traffic sample information on the network. It generalizes traffic dataset samples with specific network backgrounds into general traffic dataset samples, so as to make them applicable to a variety of network scenarios while preserving the traffic distribution pattern.

[0027] The combination process in step 3 refers to combining the network configuration module provided in step 1 with the traffic generation module provided in step 2 to generate network characteristic description files that simulate different network scenarios.

[0028] The specific process of step 5 is as follows: In a real network or network simulator, the function of building a corresponding network scenario based on the network structure and configuration file is implemented, and the sending and scheduling of traffic samples is performed during network operation. The data statistics module is used to statistically analyze the information of the corresponding queue and sending module.

[0029] The custom network simulation module comprises five sub-modules: network configuration module, node configuration module, traffic processing module, routing and forwarding module, and queue scheduling module.

[0030] Network configuration module: The network configuration module is used to configure the basic network structure, such as node locations, node scheduling strategies, link characteristics and corresponding link connections. It is the basic configuration of the entire network simulation environment.

[0031] Node Configuration Module: This module simulates the functions of device nodes in a network scenario. Each node includes a traffic processing module, a routing and forwarding module, and a queue scheduling module, responsible for traffic transmission and processing, routing, and scheduling, respectively. During the simulation, when a data packet arrives at the node, it first enters the routing and forwarding module. Based on a preset routing table, the module selects the appropriate path and output port, forwarding the data packet to the corresponding port queue. Then, according to different queue scheduling strategies, the data packet is scheduled and finally forwarded to the output port for transmission, thus enabling the node to forward data packets.

[0032] Traffic processing module: This module is responsible for two functions: traffic sending and message processing. During the simulation environment initialization phase, the traffic processing module of each node sets up traffic sending, selects traffic with its own source and destination from the provided multi-type mixed flow samples, and generates corresponding discrete data packets according to the configured traffic size, service category, and destination-related characteristics. It sets the simulation sending time for the data packets according to the sending time. After all nodes complete the traffic sending settings, the simulation environment starts running and sends the corresponding data packets at the corresponding time according to the simulation time to perform traffic replay. During the replay process, the traffic processing module processes the received data packets and collects data packet information for subsequent processing by the statistics module.

[0033] Routing and forwarding module: In the routing and forwarding module, the configured routing file is parsed to generate the routing table corresponding to each node, which is stored in the forwarding port of each destination node. During the simulation, the corresponding forwarding is performed according to the generated routing table.

[0034] Queue scheduling module: In the queue scheduling module, a scheduling queue is maintained for each port, and a set of queues is maintained in each scheduling queue. Data packets are scheduled according to different scheduling algorithms, and the queue status is statistically analyzed.

[0035] Data Statistics Module: This module is responsible for processing the statistical data from the custom network simulation module, obtaining the network-wide performance metrics and queue status, and outputting them to a data statistics file for saving.

[0036] A large-scale batch data acquisition system for network emulators includes:

[0037] The network configuration building module is used to build network configuration modules based on relevant network knowledge in the network scenario;

[0038] The traffic generation building module is used to select different traffic dataset samples from public traffic datasets based on the characteristics of traffic information in network scenarios, and to build the traffic generation module, i.e., the traffic component library.

[0039] The scenario description construction module combines components from the basic network component module according to the characteristics of different network scenarios. It selects a basic topology and routing algorithm that conforms to the characteristics of the network scenario to form an abstract network description. It selects traffic samples from the traffic generation module as traffic inputs to the abstract network to form a traffic configuration description. The abstract network description and the traffic configuration description are combined to generate the corresponding scenario description file. The above steps are repeated to quickly generate large-scale scenario description files, thus building the scenario description module, i.e., the scenario description component library.

[0040] Custom network simulation module and data statistics module are used to build custom network simulation modules and data statistics modules in real networks or network simulators;

[0041] The data metric statistics file generation module is used to simulate the scenario description file on the basis of building a custom network simulation module, and to generate a data metric statistics file by statistically analyzing the metrics through the data statistics module.

[0042] The data sample set integration module is used to integrate the scenario description file and the generated data indicator statistics file to obtain a large-scale data sample set for different network scenarios.

[0043] A large-scale batch data acquisition device for network emulators includes:

[0044] Memory: Used to store the computer program that implements the large-scale batch data acquisition method for a network simulator;

[0045] Processor: Used to implement the large-scale batch data acquisition method for a network simulator when executing the computer program.

[0046] A computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the aforementioned method for large-scale batch data acquisition for a network simulator.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0048] 1. This invention introduces components such as a basic topology component library and a traffic component library to divide network scenario functions into two parts: network configuration and traffic forwarding. The modular approach for different parts enables flexible decomposition of different components, allowing for customized configuration according to different scenario needs, thus broadening the application scope. The network configuration part introduces routing and scheduling strategies, making the simulation of network behavior more realistic and more in line with the needs of real network scenarios.

[0049] 2. Based on a network simulator, this invention customizes network function modules, implementing functional modules such as node configuration, routing and forwarding, and queue scheduling. It also employs generalization design, enabling rapid simulation of network scenarios based on scenario description files via executable scripts. Compared to existing simulation technologies, this simplifies the development process and enables rapid configuration and simulation of network simulation functions. Furthermore, it provides a method for large-scale batch data acquisition for network simulators.

[0050] 3. This invention modularizes network functions such as basic topology, routing strategies, and scheduling strategies, subdividing traditional network scenarios into network configuration modules and traffic generation modules. Functions within the network scenario are deconstructed into different modules and developed into different component libraries, facilitating flexible configuration based on various network scenarios. Furthermore, based on traditional network simulators, a configurable scenario description file is introduced to describe different scenarios, enabling rapid simulation based on the scenario description file, thus facilitating large-scale, batch network simulation.

[0051] In summary, this invention abstracts and simulates the components in a network scenario, decomposing the corresponding network scenario into different functional modules. Based on a network simulator, it enables the customization of networks according to different scenarios, and rapidly generates large-scale data samples based on different configuration information. This provides data support for model training in the field of graph neural networks, meets the needs of network modeling, and can achieve large-scale quantitative data collection. It can collect complete network data, and therefore has the characteristics of flexible use, fast generation speed, and the ability to truly reflect the network's operating mechanism. Attached Figure Description

[0052] Figure 1 This is a flowchart of the method of the present invention.

[0053] Figure 2 This is a schematic diagram of the overall framework of the present invention. Detailed Implementation

[0054] The invention will now be described in detail with reference to the accompanying drawings.

[0055] See Figure 1 A method for large-scale batch data acquisition for network simulators includes the following steps:

[0056] Step 1: Based on relevant network knowledge in the network scenario, construct a network configuration module, including a basic topology component library and a routing algorithm component library;

[0057] Step 2: Based on the traffic characteristics in the network scenario, select different traffic data samples from the public traffic dataset for processing to build a traffic generation module, i.e., a traffic component library;

[0058] Step 3: Based on the characteristics of different network scenarios, combine the components in the network configuration module built in Step 1, select the basic topology components and routing algorithm components that conform to the characteristics of the network scenario to form an abstract network description, select traffic samples from the traffic generation module built in Step 2 as traffic inputs in the abstract network to form a traffic configuration description, and combine the abstract network description and the traffic configuration description to generate the corresponding scenario description file.

[0059] Step 4: Repeat step 3 to quickly generate large-scale scene description files and build scene description modules, i.e., scene description component libraries;

[0060] Step 5: In a real network or network simulator, build a custom network simulation module and data statistics module;

[0061] Step 6: On the custom network simulation module built in Step 5, simulate the scenario description file from Step 4. Use the data statistics module to collect statistics on the metrics and generate a data metric statistics file. This mainly involves batch simulation within the custom network module built in Step 5, enabling rapid network configuration and simulation based on the scenario description file. An executable script is used to quickly assemble the custom network module based on the scenario description file, build the corresponding network scenario, and simulate based on preset traffic samples. The data statistics module processes and collects statistics on latency, packet loss, and jitter from the custom network simulation module, records the status of each port queue, and outputs the data to the data statistics file for saving.

[0062] Step 7: Integrate the scenario description file from Step 4 with the data indicator statistics file generated in Step 6 to obtain a large-scale data sample set for different network scenarios.

[0063] See Figure 2 The network configuration module in step 1 is a basic topology component library and routing algorithm component library required in a specific network scenario. It is responsible for setting and planning the basic network structure and forwarding functions in the network scenario to ensure the realization of basic network functions.

[0064] The basic topology component library mainly includes node information and link information, and its basic characteristics are described below:

[0065] Node information: includes node location, number of ports, scheduling policy node configuration information at the ports;

[0066] Link information includes link source and destination, link packet loss rate, link bandwidth, link latency, and other link characteristics.

[0067] Routing algorithm component library: Includes the planning of routing and forwarding logic in the entire network topology, configured in the form of static routing tables, and dynamically generated according to different routing algorithms.

[0068] The basic topology component library abstracts and describes common network topologies such as NSFNET and Geant, transforming the topology into a general graph structure, describing the relationships between nodes and links, and setting specific node and link information.

[0069] The routing algorithm component library solves routes for each graph structure based on common routing calculation methods such as the shortest path algorithm. It calculates the shortest hop count forwarding path for each source-destination node pair and stores it in the network routing table.

[0070] The traffic generation module in step 2 performs feature cleaning and generalization based on various publicly available real traffic sample information. While retaining the corresponding sending patterns such as the type, size, and time of the collected traffic, it generalizes some irrelevant information such as traffic content and source / destination. In the network topology generated in step 1, the traffic is replayed, and the traffic dataset samples with specific network backgrounds are generalized into general traffic dataset samples. While retaining the traffic distribution pattern, it enables the traffic to be applied to various network scenarios.

[0071] The combination process in step 3 refers to the need to maintain a one-to-one correspondence between the basic topology component library and the routing algorithm component library selected in step 1, constructing a routing and forwarding strategy under the corresponding network topology, and maintaining a correspondence between the traffic component library selected in step 2 and the basic topology component library, realizing traffic replay under the corresponding network topology. Each scenario description file corresponds to a network simulation setting under a specific network scenario, generating network characteristic description files that simulate different network scenarios.

[0072] The specific process of step 5 is as follows: In a real network or network simulator, the function of building a corresponding network scenario based on the network structure and configuration file is implemented, and the sending and scheduling of traffic samples is performed during network operation. The data statistics module is used to statistically analyze the information of the corresponding queue and sending module.

[0073] The custom network simulation module comprises five sub-modules: network configuration module, node configuration module, traffic processing module, routing and forwarding module, and queue scheduling module.

[0074] Network configuration module: The network configuration module is used to configure the basic network structure, such as node locations, node scheduling strategies, link characteristics and corresponding link connections. It is the basic configuration of the entire network simulation environment.

[0075] Node Configuration Module: This module simulates the functions of device nodes in a network scenario. Each node includes a traffic processing module, a routing and forwarding module, and a queue scheduling module, responsible for traffic transmission and processing, routing, and scheduling, respectively. During the simulation, when a data packet arrives at the node, it first enters the routing and forwarding module. Based on a preset routing table, the module selects the appropriate path and output port, forwarding the data packet to the corresponding port queue. Then, according to different queue scheduling strategies, the data packet is scheduled and finally forwarded to the output port for transmission, thus enabling the node to forward data packets.

[0076] Traffic processing module: This module is responsible for two functions: traffic sending and message processing. During the simulation environment initialization phase, the traffic processing module of each node sets up traffic sending, selects traffic with its own source and destination from the provided multi-type mixed flow samples, and generates corresponding discrete data packets according to the configured traffic size, service category, and destination-related characteristics. It sets the simulation sending time for the data packets according to the sending time. After all nodes complete the traffic sending settings, the simulation environment starts running and sends the corresponding data packets at the corresponding time according to the simulation time to perform traffic replay. During the replay process, the traffic processing module processes the received data packets and collects data packet information for subsequent processing by the statistics module.

[0077] Routing and forwarding module: In the routing and forwarding module, the configured routing file is parsed to generate the routing table corresponding to each node, which is stored in the forwarding port of each destination node. During the simulation, the corresponding forwarding is performed according to the generated routing table.

[0078] Queue scheduling module: In the queue scheduling module, a scheduling queue is maintained for each port, and a set of queues is maintained in each scheduling queue. Data packets are scheduled according to different scheduling algorithms, and the queue status is statistically analyzed.

[0079] Data Statistics Module: This module is responsible for processing the statistical data from the custom network simulation module, obtaining the network-wide performance metrics and queue status, and outputting them to a data statistics file for saving.

[0080] A large-scale batch data acquisition system for network emulators includes:

[0081] The network configuration building module is used to build network configuration modules based on relevant network knowledge in the network scenario;

[0082] The traffic generation building module is used to select different traffic dataset samples from public traffic datasets based on the characteristics of traffic information in network scenarios, and to build the traffic generation module, i.e., the traffic component library.

[0083] The scenario description construction module combines components from the basic network component module according to the characteristics of different network scenarios. It selects a basic topology and routing algorithm that conforms to the characteristics of the network scenario to form an abstract network description. It selects traffic samples from the traffic generation module as traffic inputs to the abstract network to form a traffic configuration description. The abstract network description and the traffic configuration description are combined to generate the corresponding scenario description file. The above steps are repeated to quickly generate large-scale scenario description files, thus building the scenario description module, i.e., the scenario description component library.

[0084] Custom network simulation module and data statistics module are used to build custom network simulation modules and data statistics modules in real networks or network simulators;

[0085] The data metric statistics file generation module is used to simulate the scenario description file on the basis of building a custom network simulation module, and to generate a data metric statistics file by statistically analyzing the metrics through the data statistics module.

[0086] The data sample set integration module is used to integrate the scenario description file and the generated data indicator statistics file to obtain a large-scale data sample set for different network scenarios.

[0087] A large-scale batch data acquisition device for network emulators includes:

[0088] Memory: Used to store the computer program that implements the large-scale batch data acquisition method for a network simulator;

[0089] Processor: Used to implement the large-scale batch data acquisition method for a network simulator when executing the computer program.

[0090] A computer-readable storage medium storing a computer program that, when executed by a processor, enables the implementation of the aforementioned method for large-scale batch data acquisition for a network simulator.

[0091] The most typical application scenario for large-scale batch data acquisition methods for network simulators is for simulating network datasets in the fields of machine learning and deep learning. For example, for traffic characteristics where multiple service types coexist in general network scenarios, current data acquisition methods lack large-scale datasets with network performance indicators. The datasets used cannot correspond to the traffic types existing in traditional wide area networks and lack physical meaning. To address this issue, the large-scale data generation method proposed in this invention can combine various types of real service traffic information to describe network scenarios based on basic network components. It can quickly simulate different types of traffic existing in general network scenarios and perform differentiated simulations according to different service scenarios.

[0092] Furthermore, large-scale batch data acquisition methods for network simulators can also be used to achieve large-scale batch network simulations. By using executable scripts, network scenarios can be quickly set up based on scenario description files. Using logic languages, network scenario generation based on the network simulator is achieved, avoiding the need to refactor network simulation code for different scenarios and improving simulation efficiency. During large-scale network simulation training, simulations can be performed one by one based on description files in the scenario description component library, while statistical modules are used to collect relevant metrics, enabling rapid simulation and evaluation of real network environments. Compared to traditional network simulators, this method improves simulation efficiency and scale, making it more suitable for simulating complex network environments.

Claims

1. A method for large-scale batch data acquisition for network simulators, characterized in that, Includes the following steps: Step 1: Based on relevant network knowledge in the network scenario, construct a network configuration module, including a basic topology component library and a routing algorithm component library; Step 2: Based on the traffic characteristics in the network scenario, select different traffic data samples from the public traffic dataset for processing to build a traffic generation module, i.e., a traffic component library; Step 3: Based on the characteristics of different network scenarios, combine the components in the network configuration module built in Step 1, select the basic topology components and routing algorithm components that conform to the characteristics of the network scenario to form an abstract network description, select traffic samples from the traffic generation module built in Step 2 as traffic inputs in the abstract network to form a traffic configuration description, and combine the abstract network description and the traffic configuration description to generate the corresponding scenario description file. Step 4: Repeat step 3 to quickly generate large-scale scene description files and build scene description modules, i.e., scene description component libraries; Step 5: In a real network or network simulator, build a custom network simulation module and data statistics module; Step 6: On the custom network simulation module built in step 5, simulate the scenario description file in step 4, and use the data statistics module to collect statistics on the indicators and generate a data indicator statistics file. Step 7: Integrate the scenario description file from Step 4 with the data indicator statistics file generated in Step 6 to obtain a large-scale data sample set for different network scenarios.

2. The method for large-scale batch data acquisition for network simulators according to claim 1, characterized in that, The network configuration module in step 1 is a basic topology component library and routing algorithm component library required in a specific network scenario. It is responsible for setting and planning the basic network structure and forwarding functions in the network scenario to ensure the realization of basic network functions. The basic topology component library mainly includes node information and link information, and its basic characteristics are described below: Node information: includes node location, number of ports, scheduling policy node configuration information at the ports; Link information includes link source and destination, link packet loss rate, link bandwidth, link latency, and other link characteristics. Routing algorithm component library: Includes the planning of routing and forwarding logic in the entire network topology, configured in the form of static routing tables, and dynamically generated according to different routing algorithms.

3. The method for large-scale batch data acquisition for network simulators according to claim 1, characterized in that, The traffic generation module in step 2 performs feature cleaning and generalization based on various publicly available real traffic sample information on the network. It generalizes traffic dataset samples with specific network backgrounds into general traffic dataset samples, so as to make them applicable to a variety of network scenarios while preserving the traffic distribution pattern.

4. The method for large-scale batch data acquisition for network simulators according to claim 1, characterized in that, The combination process in step 3 refers to combining the network configuration module provided in step 1 with the traffic generation module provided in step 2 to generate network characteristic description files that simulate different network scenarios.

5. The method for large-scale batch data acquisition for network simulators according to claim 1, characterized in that, The specific process of step 5 is as follows: In a real network or network simulator, the function of building a corresponding network scenario based on the network structure and configuration file is implemented, and the sending and scheduling of traffic samples is performed during network operation. The data statistics module is used to statistically analyze the information of the corresponding queue and sending module. The custom network simulation module comprises five sub-modules: network configuration module, node configuration module, traffic processing module, routing and forwarding module, and queue scheduling module. Network configuration module: The network configuration module is used to configure the basic network structure, such as node locations, node scheduling strategies, link characteristics and corresponding link connections. It is the basic configuration of the entire network simulation environment. Node Configuration Module: This module simulates the functions of device nodes in a network scenario. Each node includes a traffic processing module, a routing and forwarding module, and a queue scheduling module, responsible for traffic transmission and processing, routing, and scheduling, respectively. During the simulation, when a data packet arrives at the node, it first enters the routing and forwarding module. Based on a preset routing table, the module selects the appropriate path and output port, forwarding the data packet to the corresponding port queue. Then, according to different queue scheduling strategies, the data packet is scheduled and finally forwarded to the output port for transmission, thus enabling the node to forward data packets. Traffic processing module: This module is responsible for two functions: traffic sending and message processing. During the simulation environment initialization phase, the traffic processing module of each node sets up traffic sending, selects traffic with its own source and destination from the provided multi-type mixed flow samples, and generates corresponding discrete data packets according to the configured traffic size, service category, and destination-related characteristics. It sets the simulation sending time for the data packets according to the sending time. After all nodes complete the traffic sending settings, the simulation environment starts running and sends the corresponding data packets at the corresponding time according to the simulation time to perform traffic replay. During the replay process, the traffic processing module processes the received data packets and collects data packet information for subsequent processing by the statistics module. Routing and forwarding module: In the routing and forwarding module, the configured routing file is parsed to generate the routing table corresponding to each node, which is stored in the forwarding port of each destination node. During the simulation, the corresponding forwarding is performed according to the generated routing table. Queue scheduling module: In the queue scheduling module, a scheduling queue is maintained for each port, and a set of queues is maintained in each scheduling queue. Data packets are scheduled according to different scheduling algorithms, and the queue status is statistically analyzed. Data Statistics Module: This module is responsible for processing the statistical data from the custom network simulation module, obtaining the network-wide performance metrics and queue status, and outputting them to a data statistics file for saving.

6. A large-scale batch data acquisition system for network simulators, characterized in that, include: The network configuration building module is used to build network configuration modules based on relevant network knowledge in the network scenario; The traffic generation building module is used to select different traffic dataset samples from public traffic datasets based on the characteristics of traffic information in network scenarios, and to build the traffic generation module, i.e., the traffic component library. The scenario description construction module is used to combine the components in the basic network component module according to the characteristics of different network scenarios, select the basic topology and routing algorithm that conform to the characteristics of the network scenario to form an abstract network description, select traffic samples from the traffic generation module as traffic inputs in the abstract network to form a traffic configuration description, and combine the abstract network description and the traffic configuration description to generate the corresponding scenario description file. Repeat the above steps to quickly generate large-scale scene description files and build scene description modules, i.e., scene description component libraries; Custom network simulation module and data statistics module are used to build custom network simulation modules and data statistics modules in real networks or network simulators; The data metric statistics file generation module is used to simulate the scenario description file on the basis of building a custom network simulation module, and to generate a data metric statistics file by statistically analyzing the metrics through the data statistics module. The data sample set integration module is used to integrate the scenario description file and the generated data indicator statistics file to obtain a large-scale data sample set for different network scenarios.

7. A large-scale batch data acquisition device for network emulators, characterized in that, include: Memory: for storing a computer program that implements the large-scale batch data acquisition method for a network simulator as described in any one of claims 1-5; Processor: Used to implement the large-scale batch data acquisition method for a network simulator as described in any one of claims 1-5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of a large-scale batch data acquisition method for a network simulator as described in any one of claims 1-5.

Images