Wide area network-oriented integrated communication analog flow generation device
By building a collective communication simulation traffic generation device, the problem of difficult to balance high precision and low overhead in a wide-area network environment is solved, efficient traffic simulation and multi-data center collaborative training are realized, and multiple network simulators are adapted to improve the applicability and efficiency of the simulator.
Patent Information
- Application Number
- CN202510598409.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2025-07-25
AI Technical Summary
Existing network traffic simulators are difficult to balance high accuracy and low overhead in wide-area network environments, and lack support for high latency and low bandwidth characteristics, so they cannot simulate dynamic bandwidth fluctuations and complex traffic distribution.
A collective communication simulation traffic generation device for wide-area networks is designed, including a collective communication input construction module, a request interception module, a parameter reconstruction module and a traffic simulation module. By converting the wide-area network topology into the traditional collective communication library input format, key parameters are intercepted and reconstructed, a logical point-to-point communication list is generated, and dynamic traffic simulation is performed based on the multi-host concurrent architecture.
It achieves a balance between high precision and low overhead in a wide area network environment, improves the authenticity and efficiency of traffic generation, is adapted to a variety of network simulators, and supports collaborative training in multiple data centers.
Smart Images
Figure CN120378338A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of distributed computing and network communication, and specifically relates to a collective communication simulation traffic generation device for wide area networks. Background Art
[0002] With the rapid development of artificial intelligence technologies such as large language models, the demand for computing resources in distributed training has increased exponentially. Traditional single data center clusters can no longer meet the demand, and multi-data center collaborative training in a wide area network environment is required. However, there are the following technical problems in the current multi-data center collaborative training in a wide area network environment: Existing network traffic simulators (such as NS-3, Astra-Sim) are difficult to balance high precision and low overhead in a wide area network environment, and there is a problem that the large-scale simulation takes significantly longer due to the discrete event-driven mechanism; Traditional collective communication libraries (such as NCCL, HCCL) are only adapted to single machine or single server environments and lack primitive support for the characteristics of high latency and low bandwidth in a wide area network environment; Existing network traffic simulators cannot simulate the dynamic bandwidth fluctuations and complex traffic distribution characteristics of wide area networks, resulting in insufficient authenticity of traffic generation. Summary of the Invention
[0003] In view of the above deficiencies in the prior art, the present invention provides a collective communication simulation traffic generation device for wide area networks.
[0004] In order to achieve the above invention objective, the technical solution adopted by the present invention is:
[0005] A collective communication simulation traffic generation device for wide area networks, comprising a collective communication input construction module, a collective communication request interception module, a collective communication parameter reconstruction module, and a collective communication traffic simulation module that are connected in sequence;
[0006] The collective communication input construction module is used to construct the collective communication input of the wide area network into the input format of the traditional collective communication library to obtain the reconstructed collective communication input;
[0007] The traditional collective communication library module is used to generate traditional collective communication library function call instructions according to the reconstructed collective communication input;
[0008] The collective communication request interception module is used to intercept the traditional collective communication library function call instructions and extract the key parameters therein;
[0009] The collective communication parameter reconstruction module is used to reconstruct the key parameters in the traditional collective communication library function call instructions to obtain a logical point-to-point communication list;
[0010] The collective communication traffic simulation module is used to decompose the logical point-to-point communication list into multiple end-to-end communication flows, generate dynamic traffic based on the multiple end-to-end communication flows through periodic bandwidth reset and distributed traffic models, and perform multi-concurrent traffic simulation of the wide area network based on the multi-host concurrent architecture.
[0011] Furthermore, the collective communication input construction module includes a network awareness sub-module, a wide area topology abstraction sub-module, and an adaptation layer construction sub-module connected in sequence; the network awareness sub-module is used to construct a network state matrix according to the state data of the wide area network; the wide area topology abstraction sub-module is used to transform the physical topology in the wide area network into a hierarchical logical topology by using a hierarchical clustering algorithm according to the network state matrix; the adaptation layer construction sub-module is used to generate an input configuration adapted to the traditional collective communication library according to the hierarchical logical topology in the wide area network, and construct the collective communication input of the wide area network into the input format of the traditional collective communication library by using the input configuration adapted to the traditional collective communication library, so as to obtain the collective communication reconstruction input.
[0012] Furthermore, in the adaptation layer construction sub-module, the process of generating an input configuration adapted to the traditional collective communication library according to the hierarchical logical topology in the wide area network is as follows: according to the hierarchical logical topology in the wide area network, communication groups of the traditional collective communication library are generated according to the bandwidth / delay selection algorithm within the layer, and hierarchical communication is performed through backbone nodes between the layers to generate an input configuration adapted to the traditional collective communication library.
[0013] Furthermore, the collective communication request interception module includes a dynamic hijacking setting sub-module, a request interception sub-module, and a data extraction sub-module connected in sequence; the dynamic hijacking setting sub-module is used to load a custom interception library through dynamic link hijacking to overwrite the function symbols of the traditional collective communication library; the request interception sub-module is used to intercept the collective communication library function call instructions according to the custom interception library and intercept its key parameters; the data extraction sub-module is used to extract the key parameters of the traditional collective communication library function call instructions.
[0014] Furthermore, the collective communication parameter reconstruction module includes a parameter conversion rule setting sub-module and a logical communication list generation sub-module connected in sequence; the parameter conversion rule setting sub-module is used to map the key parameters of the traditional collective communication library functions to the logical identifiers of the collective communication traffic simulation module; the logical communication list generation sub-module is used to generate a logical point-to-point communication list according to the logical identifiers of the module collective communication traffic simulation module.
[0015] Furthermore, the information of the logical communication list operation includes the source address, destination address, data volume, and dependency relationship.
[0016] Further, in the collective communication traffic simulation module, the logical point-to-point communication list is decomposed into multiple end-to-end communication flows, and dynamic traffic is generated based on the multiple end-to-end communication flows through periodic bandwidth reset and distributed traffic models. The specific process is as follows: Based on Socket communication and combined with the Iperf3 bandwidth test tool, through a preset encapsulation method, the logical point-to-point communication list is decomposed into multiple end-to-end communication flows, and traffic packets of different sizes and protocols are generated in real time according to the multiple end-to-end communication flows. The packet sending interval, packet size, and network bandwidth fluctuation during the sending of traffic packets are controlled to generate dynamic traffic.
[0017] The beneficial effects of the present invention are as follows:
[0018] (1) By constructing the collective communication input construction module, the multi-center topology of the wide area network is converted into the input format of the traditional collective communication library. Then, through the constructed collective communication request interception module, the collective communication library functions are called and the key parameters of the collective communication library functions are extracted. Finally, through the constructed collective communication parameter reconstruction module, the key parameters of the collective communication library functions are reconstructed to construct a logical point-to-point communication list, which can fill the gap in the collective communication primitive library of multi-data centers in the wide area network.
[0019] (2) By constructing the collective communication traffic simulation module, the logical point-to-point communication list is decomposed into multiple end-to-end communication flows. Based on the multiple end-to-end communication flows, dynamic traffic is generated through periodic bandwidth reset and distribution models, and large-scale traffic simulation is realized based on the multi-homed host concurrent architecture, which can improve the simulation authenticity of the traffic generator for the dynamic characteristics of the wide area network.
[0020] (3) By constructing the collective communication request interception module, the custom interception library is loaded through dynamic link hijacking to overwrite the symbols of the traditional collective communication library functions. Then, according to the custom interception library, the call instructions of the collective communication library functions are intercepted, the key parameters are intercepted and extracted, and a logical point-to-point communication list is generated through the constructed collective communication parameter reconstruction module, avoiding the full-scale simulation of the underlying communication details by the existing network traffic simulator, and only simulating the key parameters in the logical communication list rather than all the network protocol stack behaviors, reducing the consumption of computing resources. Then, through the constructed collective communication traffic simulation module, the complex collective communication tasks are decomposed into multiple end-to-end communication flows, and dynamic traffic is generated through periodic bandwidth reset and distributed traffic models, which can accurately simulate the network behavior of each flow, avoid the high overhead of full-scale packet-level simulation, and use multi-homed host concurrent control for concurrent traffic generation, effectively solving the problem that it is difficult to balance high precision and low overhead of the network simulator in the wide area network environment. Description of the Drawings
[0021] Figure 1Schematic structural diagram of a collective communication simulation traffic generation device for a wide area network;
[0022] Figure 2 Schematic diagram of the form for reconstructing the input of collective communication;
[0023] Figure 3 Schematic example diagram of a logical point-to-point communication list;
[0024] Figure 4 Schematic diagram of the multi-host concurrent architecture of the collective communication traffic simulation module. Detailed implementation manner
[0025] The following describes the detailed implementation manner of the present invention to facilitate those skilled in the art of the present technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the detailed implementation manner. For those of ordinary skill in the art of the present technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.
[0026] As Figure 1 shown, a collective communication simulation traffic generation device for a wide area network includes a collective communication input construction module, a traditional collective communication library module, a collective communication request interception module, a collective communication parameter reconstruction module, and a collective communication traffic simulation module that are connected in sequence.
[0027] In an alternative embodiment of the present invention, the collective communication input construction module is configured to construct the collective communication input of a wide area network into the input format of a traditional collective communication library to obtain a collective communication reconstruction input. The collective communication input construction module includes a network awareness sub-module, a wide area topology abstraction sub-module, and an adaptation layer construction sub-module that are connected in sequence; the network awareness sub-module is configured to construct a network state matrix based on the state data of the wide area network; the wide area topology abstraction sub-module is configured to convert the physical topology in the wide area network into a hierarchical logical topology by using a hierarchical clustering algorithm according to the network state matrix; the adaptation layer construction sub-module is configured to generate an input configuration adapted to the traditional collective communication library according to the hierarchical logical topology in the wide area network, and construct the collective communication input of the wide area network into the input format of the traditional collective communication library by using the input configuration adapted to the traditional collective communication library to obtain a collective communication reconstruction input. In the adaptation layer construction sub-module, the process of generating an input configuration adapted to the traditional collective communication library according to the hierarchical logical topology in the wide area network is as follows: according to the hierarchical logical topology in the wide area network, a communication group of the traditional collective communication library is generated by selecting an algorithm (Tree / Ring-AllReduce) according to bandwidth / delay within the layer, and hierarchical communication is performed through backbone nodes between the layers to generate an input configuration adapted to the traditional collective communication library. The bandwidth / delay selection algorithm in the present invention adopts Tree / Ring-AllReduce.
[0028] As Figure 2 shown, the present invention uses a two-dimensional matrix to show the form of the collective communication reconstruction input. Each element is a tuple, and the meanings of the tuple are, in sequence: bandwidth, jitter correlation rate, delay jitter distribution, link delay, delay jitter, link packet loss rate, and queue size. The matrix elements below represent the link configuration information from node to node.
[0029] Specifically, since the traditional collective communication library is only adapted to collective communication between single machines with multiple cards / single servers with multiple machines, and its input is also designed for the above scenarios. Therefore, the present invention designs a collective communication input construction module, which can construct the collective communication input of a wide area network into the input format of a traditional collective communication library to obtain a collective communication reconstruction input.
[0030] The working principle of the collective communication input construction module is as follows:
[0031] The network awareness sub-module inputs the status data of the wide-area network, including the node IP list and preset connection relationships of the wide-area network, and then actively probes through Ping / iPerf and passively listens through SNMP / NetFlow to collect real-time link delay, bandwidth, and packet loss rate. Finally, it outputs a network status matrix, which includes delay, bandwidth, and packet loss rate. The wide-area topology abstraction sub-module inputs the network status matrix, then divides the low-delay area based on the Prim algorithm, constructs the minimum spanning tree within the area, and connects across domains through backbone nodes. Finally, it outputs a hierarchical logical topology, which includes area division data and backbone connection data. The adaptation layer construction sub-module inputs the hierarchical logical topology, then selects an algorithm (Tree / Ring-AllReduce) for bandwidth / delay within the layer to generate communication groups for the traditional collective communication library, and performs hierarchical communication through backbone nodes between layers to output the input configuration adapted to the traditional collective communication library.
[0032] In an optional embodiment of the present invention, the traditional collective communication library module is used to generate traditional collective communication library function call instructions according to the collective communication reconstruction input. The traditional collective communication library module adopts NCCL (NVIDIA Collective Communications Library, a traditional collective communication library) or HCCL (Huawei Collective Communications Library, a traditional collective communication library).
[0033] Specifically, the working principle of the traditional collective communication library module is as follows:
[0034] The traditional collective communication library module receives the collective communication reconstruction input, then selects a traditional collective communication library function according to the data volume and hardware topology, and finally converts the selected traditional collective communication library function into a low-level call, binding the data buffer, communication domain, and synchronization mechanism to generate traditional collective communication library function call instructions.
[0035] In an alternative embodiment of the present invention, the collective communication request interception module is used to intercept the traditional collective communication library function call instructions and extract the key parameters therein. The collective communication request interception module includes a dynamically hijacking setting sub-module, a request interception sub-module, and a data extraction sub-module connected in sequence; the dynamically hijacking setting sub-module is used to load a custom interception library through dynamic link hijacking to overwrite the traditional collective communication library function symbols; the request interception sub-module is used to intercept the collective communication library function call instructions according to the custom interception library and intercept its key parameters; the data extraction sub-module is used to extract the key parameters of the traditional collective communication library function call instructions. In the dynamically hijacking setting sub-module, the dynamic link hijacking adopts LD_PRELOAD. LD_PRELOAD is a dynamic link control tool under Linux. In the present invention, the custom interception library is loaded by adopting LD_PRELOAD to overwrite the traditional collective communication library function symbols.
[0036] Specifically, the working principle of the collective communication request interception module is as follows:
[0037] The dynamically hijacking setting sub-module sets the environment variable. By using LD_PRELOAD = / path / to / intercept_lib.so, the program is forced to load the custom interception library first. Then, the traditional collective communication library functions are defined in the interception library for symbol overwriting, and then the traditional collective communication library functions are replaced with the function addresses in the interception library in the dynamic link hijacking to ensure that the function calls are hijacked; in the request interception sub-module, when it is detected that the target function is called, it immediately triggers interception according to the custom interception library to complete the interception of the collective communication library function call instructions, and intercepts its key parameters from the collective communication library function call instructions, including communication data description, communication participants, and execution context, and returns a call success signal to the traditional collective communication library to avoid actual data transmission; then, the key parameters in the collective communication library function call instructions are extracted in the data extraction sub-module.
[0038] In an alternative embodiment of the present invention, the collective communication parameter reconstruction module is used to reconstruct the key parameters in the traditional collective communication library function call instructions to obtain a logical point-to-point communication list. The collective communication parameter reconstruction module includes a parameter conversion rule setting sub-module and a logical communication list generation sub-module connected in sequence; the parameter conversion rule setting sub-module is used to map the key parameters of the traditional collective communication library to the logical identifiers of the collective communication traffic simulation module; the logical communication list generation sub-module is used to generate a logical point-to-point communication list according to the logical identifiers of the collective communication traffic simulation module. The information of the logical communication list operation includes source address, destination address, data volume, and dependency relationship.
[0039] Such as Figure 3As shown, the present invention shows the point-to-point communication list after mapping the collective communication operation in the format of Json. Each id (corresponding to Figure 4 "1" and "2" in it) corresponds to a point-to-point communication operation, and the information of the operation includes the source address src, the destination address dst, the data volume data_size, and the dependency relationship dependency. Among them, the dependency relationship represents the serial order. That is: if the dependency of B = "A", it is considered that B can and can only start execution after the execution of operation A ends, thus defining the point-to-point communication list.
[0040] Specifically, the working principle of the collective communication parameter reconstruction module is as follows:
[0041] The parameter conversion rule setting sub-module first pre-configures the IP-Port mapping table, then obtains the Rank number of the key parameters of the traditional collective communication library function, and queries the pre-configured IP-Port mapping table using the Rank number of the key parameters of the traditional collective communication library function to map the key parameters of the traditional collective communication library function to the logical identifier of the collective communication traffic simulation module; the logical communication list generation sub-module obtains the Socket binding information using the netstat or ss command according to the logical identifier of the module for simulating collective communication traffic, then combines the container network namespace, locates the logical communication IP, and generates a logical communication list using the logical communication IP.
[0042] In an optional embodiment of the present invention, the collective communication traffic simulation module is used to decompose the logical point-to-point communication list into multiple end-to-end communication flows, generate dynamic traffic based on the multiple end-to-end communication flows through periodic bandwidth reset and distributed traffic models, and perform multi-concurrent traffic simulation of the wide area network based on the multi-homed host concurrent architecture, as Figure 4 shown. In the collective communication traffic simulation module, decomposing the logical point-to-point communication list into multiple end-to-end communication flows and generating dynamic traffic based on the multiple end-to-end communication flows through periodic bandwidth reset and distributed traffic models, the specific process is as follows: Based on Socket communication and combined with the Iperf3 bandwidth test tool, through a preset encapsulation method, decompose the logical point-to-point communication list into multiple end-to-end communication flows, generate traffic packets of different sizes and protocols in real time according to the multiple end-to-end communication flows, and control the packet sending interval, packet size, and network bandwidth fluctuation when sending the traffic packets to generate dynamic traffic.
[0043] Those of ordinary skill in the art will realize that the embodiments described herein are provided to assist the reader in understanding the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not depart from the essence of the present invention based on these technical revelations disclosed in the present invention, and these deformations and combinations are still within the scope of protection of the present invention.
Claims
1. An aggregate communication simulation traffic generation device for a wide area network, characterized in that It includes a collective communication input construction module, a traditional collective communication library module, a collective communication request interception module, a collective communication parameter reconstruction module, and a collective communication traffic simulation module that are connected in sequence; The collective communication input construction module is used to construct the collective communication input of the wide area network into the input format of the traditional collective communication library to obtain the collective communication reconstruction input; The traditional collective communication library module is used to generate traditional collective communication library function call instructions according to the collective communication reconstruction input; The collective communication request interception module is used to intercept the traditional collective communication library function call instructions and extract the key parameters therein; The collective communication parameter reconstruction module is used to reconstruct the key parameters in the traditional collective communication library function call instructions to obtain a logical point-to-point communication list; The collective communication traffic simulation module is used to decompose the logical point-to-point communication list into multiple end-to-end communication flows, generate dynamic traffic based on the multiple end-to-end communication flows through periodic bandwidth reset and distributed traffic models, and perform multi-concurrent traffic simulation of the wide area network based on the multi-homed host concurrent architecture.
2. The collective communication simulation traffic generation device for a wide area network according to claim 1, characterized in that The collective communication input construction module includes a network awareness sub-module, a wide area topology abstraction sub-module, and an adaptation layer construction sub-module that are connected in sequence; The network awareness sub-module is used to construct a network state matrix according to the state data of the wide area network; The wide area topology abstraction sub-module is used to transform the physical topology in the wide area network into a hierarchical logical topology using a hierarchical clustering algorithm according to the network state matrix; The adaptation layer construction sub-module is used to generate an input configuration that adapts to the traditional collective communication library according to the hierarchical logical topology in the wide area network, and use the input configuration that adapts to the traditional collective communication library to construct the collective communication input of the wide area network into the input format of the traditional collective communication library to obtain the collective communication reconstruction input.
3. The set communication simulation traffic generation device for wide area network according to claim 2, characterized in that, In the adaptation layer construction sub-module, generating an input configuration that adapts to the traditional collective communication library according to the hierarchical logical topology in the wide area network, the specific process is as follows: according to the hierarchical logical topology in the wide area network, generate communication groups of the traditional collective communication library according to the bandwidth / delay selection algorithm within the layer, and perform hierarchical communication through backbone nodes between the layers to generate an input configuration that adapts to the traditional collective communication library.
4. The set communication simulation traffic generation device for a wide area network according to claim 1, characterized in that The collective communication request interception module includes a dynamic hijacking setting sub-module, a request interception sub-module, and a data extraction sub-module that are connected in sequence; the dynamic hijacking setting sub-module is used to load a custom interception library through dynamic link hijacking to overwrite the traditional collective communication library function symbols; The request interception sub-module is used to intercept the collective communication library function call instructions according to the custom interception library and intercept its key parameters; The data extraction sub-module is used to extract the key parameters of the traditional collective communication library function call instructions.
5. The collective communication simulation traffic generation device for a wide area network according to claim 1, characterized in that, The collective communication parameter reconstruction module includes a parameter conversion rule setting sub-module and a logical communication list generation sub-module that are connected in sequence; the parameter conversion rule setting sub-module is used to map the key parameters of the traditional collective communication library function to the logical identifiers of the collective communication traffic simulation module; the logical communication list generation sub-module is used to generate a logical point-to-point communication list according to the logical identifiers of the collective communication traffic simulation module.
6. The collective communication simulation traffic generation device for a wide area network according to claim 5, wherein The information of logical communication list operations includes source address, destination address, data volume, and dependencies.
7. The set communication simulation traffic generation device for wide area network according to claim 1, characterized in that In the collective communication traffic simulation module, the logical point-to-point communication list is decomposed into multiple end-to-end communication flows. Based on multiple end-to-end communication flows, dynamic traffic is generated through periodic bandwidth reset and distributed traffic model. The specific process is as follows: Based on Socket communication and combined with the Iperf3 bandwidth testing tool, through a preset encapsulation method, the logical point-to-point communication list is decomposed into multiple end-to-end communication flows. According to multiple end-to-end communication flows, traffic packets of different sizes and protocols are generated in real time, and the packet sending interval, packet size, and network bandwidth fluctuation during the sending of traffic packets are controlled to generate dynamic traffic.