Traffic monitoring method and apparatus

Abstract

The application discloses a traffic monitoring method and device, the method comprises the following steps: establishing a communication connection based on multiple GPUs in a communication domain; creating a shared memory according to the identification of the communication domain, and allocating corresponding memory spaces for the multiple GPUs to store the communication traffic statistical information of each GPU; mapping the address of the communication traffic statistical information to the process address space by using the memory mapping, and storing the mapped address to a first structure; wherein the structure points to the mapped address by using a pointer; when data access is performed between the multiple GPUs, the GPU stores the communication traffic statistical information to the address of the memory space, and updates the pointer of a second structure by using the pointer of the first structure, so that the traffic receiving information and / or the traffic sending information of each GPU are obtained according to the second structure. The shared memory stores the communication traffic statistical information, the stored address is mapped and stored to the pointer of the first structure, and the data access is updated to the second structure, so that the traffic monitoring of each GPU in the communication domain is realized.

Description

Technical Field

[0001] This application relates to the field of Internet technology, specifically to a traffic monitoring method and device. Background Technology

[0002] Large-scale language models, also known as large-scale language models, are capable of learning more subtle patterns and rules, exhibiting stronger generalization and expressive abilities. They possess a massive number of parameters, and training and inference require robust infrastructure capabilities, such as ChatGPT. Since a single machine cannot accommodate such a large number of parameters, distributed machine clusters are typically used for training and inference, with high-performance networks and ensemble communication being crucial components.

[0003] In existing technologies, high-performance network communication can be achieved using techniques such as RDMA (Remote Direct Memory Access), which bypasses remote memory access by the operating system kernel, offering significant advantages in latency, throughput, and CPU consumption. Collective communication libraries, such as NVIDIA's open-source NCCL (NVIDIA Collective Communications Library), facilitate the exchange of memory data between multiple GPUs. However, the NCCL library lacks monitoring for GPU collective communication. When GPUs communicate simultaneously through multiple RDMA network cards, traffic within the same communication domain is distributed across different RDMA network cards, making it impossible to distinguish traffic within a specific communication domain based on RDMA network card traffic. Similarly, when large model training uses parallel techniques—where a single GPU concurrently performs different data processing in different communication domains—it's impossible to differentiate traffic across multiple parallel communication domains, hindering timely detection of training bottlenecks. Summary of the Invention

[0004] In view of the above problems, embodiments of this application are proposed to provide a traffic monitoring method and apparatus that overcomes or at least partially solves the above problems.

[0005] According to a first aspect of the embodiments of this application, a traffic monitoring method is provided, comprising:

[0006] Establish communication connections based on multiple GPUs in the communication domain;

[0007] Shared memory is created based on the identifier of the communication domain, and corresponding memory space is allocated for multiple GPUs to store communication traffic statistics for each GPU; the communication traffic statistics include traffic reception information and / or traffic transmission information;

[0008] The address of communication traffic statistics is mapped to the process address space using memory mapping, and the mapped address is stored in the first structure of the GPU. The first structure is initialized when the communication connection is established, and the first structure is pointed to by a pointer to the mapped address.

[0009] When multiple GPUs access data, the GPU stores communication traffic statistics to the corresponding memory address and updates the pointer of the second structure of the communication request using the pointer of the first structure, so as to obtain the traffic reception information and / or traffic transmission information of each GPU according to the second structure.

[0010] Optionally, the multiple GPUs include a sending GPU and a receiving GPU; the first structure includes a receiving structure and / or a sending structure;

[0011] Establishing a communication connection based on multiple GPUs in the communication domain further includes:

[0012] The receiving GPU listens for connection requests. When it receives a connection request from the sending GPU, the receiving GPU accepts the connection request and establishes a communication connection. The receiving GPU creates a receive structure, and the sending GPU creates a send structure. The sending GPU and the receiving GPU belong to the same communication domain.

[0013] Optionally,

[0014] The length of the memory space is a preset byte length; the communication numbers of the sending GPU and the receiving GPU are different;

[0015] Shared memory is created based on the identifier of the communication domain, and corresponding memory space is allocated for multiple GPUs to store communication traffic statistics for each GPU. This further includes:

[0016] Create shared memory based on the identifier of the communication domain;

[0017] The offset of the memory space is calculated based on the communication number of the receiving GPU and the sending GPU to determine the starting address for allocating memory space for each GPU.

[0018] The receiving GPU stores the received traffic information, and the sending GPU stores the sent traffic information.

[0019] Optionally, using memory mapping to map the address of communication traffic statistics to the process address space, and storing the mapped address in the GPU's first structure further includes:

[0020] Memory mapping is used to map the addresses of traffic receiving information of each GPU to the process address space, and the mapped addresses of traffic receiving information are stored in the receiving structure of the receiving GPU.

[0021] Memory mapping is used to map the addresses of traffic transmission information of each GPU to the process address space, and the mapped addresses of traffic transmission information are stored in the transmission structure of the sending GPU.

[0022] Optionally, the method further includes:

[0023] Add a receive pointer and a send pointer to the second structure.

[0024] Optionally, when multiple GPUs access data, the GPU stores communication traffic statistics to the corresponding memory address, and updates the pointer of the second structure of the communication request using the pointer of the first structure, so as to obtain the traffic reception information and / or traffic transmission information of each GPU according to the second structure. This further includes:

[0025] When the receiving GPU accesses the data of the sending GPU, it initiates a receive request and passes a receive structure, and updates the receive pointer in the second structure according to the pointer of the receive structure; the sending GPU initiates a send request and passes a send structure, and updates the send pointer in the second structure according to the pointer of the send structure.

[0026] The traffic reception information and / or traffic transmission information of each GPU are determined based on the receive pointer and / or send pointer of the second structure.

[0027] Optionally, obtaining the traffic reception information and / or traffic transmission information of each GPU according to the second structure further includes:

[0028] The receive and / or send information of different GPUs in the communication domain is determined based on the receive pointer and / or send pointer of the second structure;

[0029] Alternatively, based on the receive pointer and / or send pointer of the second structure, the total traffic received information and / or total traffic sent information of multiple GPUs under the same distributed node can be counted.

[0030] Alternatively, based on the receive pointers and / or send pointers of different second structures, statistical analysis can be performed on traffic reception and / or traffic transmission information belonging to different communication domains within the same GPU.

[0031] Optionally, the method further includes:

[0032] The communication links of multiple GPUs in the communication domain are pre-configured so that multiple GPUs can execute according to the communication links; the communication links include establishing communication connections, creating shared memory, allocating memory space, storing communication traffic statistics, and mapping addresses to the first structure of the GPU.

[0033] According to a second aspect of the embodiments of this application, a traffic monitoring device is provided, comprising:

[0034] A connection module is established, suitable for establishing communication connections between multiple GPUs in a communication domain;

[0035] A shared memory module is suitable for creating shared memory based on the identifier of a communication domain and allocating corresponding memory space for multiple GPUs to store communication traffic statistics for each GPU; the communication traffic statistics include traffic reception information and / or traffic transmission information.

[0036] The address storage module is adapted to use memory mapping to map the address of communication traffic statistics information to the process address space, and store the mapped address in the first structure of the GPU; wherein, the first structure is initialized when the communication connection is established, and the first structure points to the mapped address with a pointer;

[0037] The traffic acquisition module is suitable for situations where multiple GPUs access data. The GPU stores communication traffic statistics to the address of the corresponding memory space and updates the pointer of the second structure of the communication request using the pointer of the first structure, so as to obtain the traffic reception information and / or traffic transmission information of each GPU according to the second structure.

[0038] According to a third aspect of the embodiments of this application, a computing device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0039] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the above-described traffic monitoring method.

[0040] According to a fourth aspect of the present application, a computer storage medium is provided, the storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the traffic monitoring method described above.

[0041] According to the traffic monitoring method and apparatus provided in this application, based on the shared memory storage of GPU communication traffic statistics in the communication domain, the address of the communication traffic statistics is mapped and stored in the pointer of the first structure. When accessing data, the first structure is used to update the second structure. Based on the second structure, traffic monitoring of the communication domain can be realized from the granularity of the communication domain, realizing traffic monitoring when multiple tasks are parallel, and further helping to monitor and evaluate the network performance and health of distributed clusters used for large model training, etc.

[0042] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0043] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0044] Figure 1 A flowchart of a traffic monitoring method according to an embodiment of this application is shown;

[0045] Figure 2 A flowchart of a traffic monitoring method according to another embodiment of this application is shown;

[0046] Figure 3a A schematic diagram illustrating the traffic monitoring performed according to this application is shown;

[0047] Figure 3b A schematic diagram illustrating traffic statistics for the same node is shown;

[0048] Figure 3c This diagram illustrates the traffic statistics for multitasking on a GPU.

[0049] Figure 4 A schematic diagram of the structure of a traffic monitoring device according to an embodiment of this application is shown;

[0050] Figure 5 A schematic diagram of the structure of a computing device according to an embodiment of this application is shown. Detailed Implementation

[0051] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0052] First, the terms and concepts involved in one or more embodiments of this application will be explained.

[0053] ChatGPT: A dialogue generation model based on natural language processing technology, which can be used in applications such as chatbots.

[0054] Large models: Deep learning models with a large number of parameters usually require a lot of computing resources and time to train.

[0055] Distributed computing: A computing method that distributes tasks to multiple computers or processors for parallel processing.

[0056] High-performance networks: Computer networks with high transmission speeds and large bandwidths.

[0057] Aggregator communication: In distributed computing, the process of aggregating the computation results of multiple computers or processors.

[0058] NCCL: NVIDIA Collective Communications Library, a high-performance collective communications library specifically designed for distributed computing of deep learning models.

[0059] RDMA: Remote Direct Memory Access, a high-performance network transmission technology that can be used to accelerate data transmission in distributed computing.

[0060] GPU: Graphics Processing Unit, which can be used for computationally intensive tasks such as training and inference of deep learning models.

[0061] Communicator: In NCCL, each GPU needs to create a communicator for communication. When a communicator is created, a unique rank communication number between 0 and n-1 is assigned to each GPU device.

[0062] Communication domain: A collection of multiple GPUs with communicators communicating with each other. In NCCL, a communication domain is identified by a unique ID. All GPU devices within the same communication domain have the same ID, and only GPUs in the same communication domain can communicate with each other.

[0063] Figure 1 A flowchart of a traffic monitoring method according to an embodiment of this application is shown, as follows: Figure 1 As shown, the method includes the following steps:

[0064] Step S101: Establish communication connections based on multiple GPUs in the communication domain.

[0065] This embodiment is based on multiple GPUs in a distributed cluster. Multiple GPUs located on the same distributed node (i.e., the same machine) can directly access data, while multiple GPUs located on different distributed nodes (i.e., multiple machines) need to access data through RDMA.

[0066] Multiple GPUs can be divided into different communication domains. For example, GPUs may include GPU1, GPU2, GPU3, and GPU4, and communication domains may include Communication Domain 1, Communication Domain 2, and Communication Domain 3. Communication Domain 1 can include GPU1 and GPU2, Communication Domain 2 can include GPU1 and GPU3, and Communication Domain 3 can include GPU1, GPU2, GPU3, and GPU4, etc. Within each communication domain, a communication number can be set for each GPU. For example, in Communication Domain 1, GPU1's communication number is rank0, and GPU2's is rank1; in Communication Domain 2, GPU1's is rank2, and GPU3's is rank3; in Communication Domain 3, GPU1's is rank4, GPU2's is rank5, GPU3's is rank6, and GPU4's is rank7. This facilitates distinguishing different GPUs located in different communication domains. The above is an example; the specific settings should be configured according to the implementation situation and are not limited here.

[0067] Based on the communication domain, communication connections can be established for multiple GPUs belonging to the same communication domain. For example, by using the API in the NCCL collection communication library, connection requests from GPUs in the communication domain can be listened to, and communication connections between GPUs can be established after accepting the connection requests.

[0068] Step S102: Create shared memory based on the identifier of the communication domain and allocate corresponding memory space for multiple GPUs to store communication traffic statistics of each GPU.

[0069] Each communication domain has a unique ID. Based on these IDs, shared memory is created for each communication domain. This shared memory can be shared by multiple GPUs on the same distributed node; that is, shared memory is set up on the same machine to store the communication traffic generated when multiple GPUs on the same machine execute tasks within the communication domain. It allows for fast read and write operations without data copying, resulting in higher efficiency. One communication domain can execute one task. For a large project, multiple tasks can be executed in parallel by different communication domains to complete the task collaboratively.

[0070] After creating shared memory, corresponding memory spaces can be allocated sequentially to multiple GPUs. For example, with two GPUs, a certain amount of memory space can be allocated to each GPU. This memory space can store communication traffic statistics generated by the GPUs during execution, including traffic reception and transmission information. When a GPU receives data, it generates traffic reception information, which can be stored in the memory space; similarly, when a GPU sends data, it generates traffic transmission information, which can also be stored in the memory space.

[0071] Step S103: Use memory mapping to map the address of communication traffic statistics information to the process address space, and store the mapped address in the first structure of the GPU.

[0072] Memory mapping methods such as mmap can be used to map addresses storing communication traffic statistics to the process address space. The mapped addresses can then be stored in the GPU's first structure. This first structure is created during GPU initialization when establishing a communication connection and can be a communication structure used in the actual communication process. A pointer is added to this first structure to point to the mapped address, allowing for easy retrieval of the stored communication traffic statistics generated during GPU data access by directly determining the address from the pointer.

[0073] The above steps S101-S103 can utilize the NCCL collection communication library to pre-configure the communication links of multiple GPUs in the communication domain, so that multiple GPUs can execute according to the configured communication links, such as establishing communication connections, creating shared memory, allocating memory space, storing communication traffic statistics, and mapping addresses to store in the first structure of the GPU. Each step can be completed using the NCCL API interface, which can be completed during the communication connection initialization phase to ensure that performance overhead is minimized.

[0074] Step S104: When multiple GPUs access data, the GPU stores the communication traffic statistics information to the address of the corresponding memory space, and updates the pointer of the second structure of the communication request using the pointer of the first structure, so as to obtain the traffic reception information and / or traffic transmission information of each GPU according to the second structure.

[0075] When multiple GPUs access each other's data, information gaps exist in the interface APIs of RDMA, making it impossible to directly transfer GPU and task-related information between APIs. Considering these issues, this embodiment modifies the second structure, such as the request structure used in actual communication, by adding pointers to the second structure. During GPU data access, the pointers in the first GPU structure can be updated to the pointers in the second structure, enabling the transfer of communication traffic information between the GPUs.

[0076] Specifically, when GPUs access data, when APIs such as receiving data and sending data are called, the pointer in the first structure of the GPU is used to update the pointer in the second structure. Thus, the flow receiving information and flow sending information of the GPU execution process stored in the shared memory can be directly obtained based on the pointer in the second structure, so as to realize the flow monitoring of each GPU in the communication domain.

[0077] This embodiment adds pointers to the first structure of the GPU and the second structure during the communication process. Communication traffic information is transmitted based on these pointers without modifying the API, ensuring API compatibility when using the NCCL collection communication library. Furthermore, traffic monitoring of the communication domains enables traffic monitoring in different communication domains when multiple tasks are executed in parallel across multiple domains. This facilitates overall multi-task traffic monitoring and analysis. For large model training on distributed clusters, it allows for timely identification of training bottlenecks and monitoring of network traffic transmission performance and health status during training, thereby improving training efficiency.

[0078] According to the traffic monitoring method provided in this application, the communication traffic statistics of the GPU in the communication domain are stored in shared memory. The address of the communication traffic statistics is mapped and stored in the pointer of the first structure. When accessing data, the first structure is used to update the second structure. Based on the second structure, traffic monitoring of the communication domain can be realized from the granularity of the communication domain. This enables traffic monitoring when multiple tasks are running in parallel, and further helps to monitor and evaluate the network performance and health of the distributed cluster when used for large model training.

[0079] Figure 2 A flowchart of a traffic monitoring method according to an embodiment of this application is shown, as follows: Figure 2 As shown, the method includes the following steps:

[0080] Step S201: The receiving GPU listens for connection requests. When it receives a connection request initiated by the sending GPU, the receiving GPU accepts the connection request and establishes a communication connection. The receiving GPU creates a receiving structure, and the sending GPU creates a sending structure.

[0081] When GPUs establish a communication connection, they are divided into receiving GPUs and sending GPUs within the same communication domain, based on the direction of data transmission and reception. The sending GPU and the receiving GPU belong to the same communication domain.

[0082] Specifically, before establishing a communication connection, both the receiving GPU and the sending GPU create their own communication structures, such as the `ncclComm` structure. The receiving GPU listens for connection requests through a listening interface. When it receives a connection request initiated by the sending GPU based on the connection interface, the receiving GPU accepts the connection request based on the receiving interface and establishes a communication connection. The receiving GPU creates a receiving structure `RecvComm`, and the sending GPU creates a sending structure `SendComm`. The receiving structure is stored in the receiving GPU's `ncclComm` structure, and the sending structure is stored in the sending GPU's `ncclComm` structure. The first structure includes the receiving structure `RecvComm` and the sending structure `SendComm`.

[0083] A GPU can be a physical GPU device, such as GPU card 1 in a server, or a communication device indicated by a communication number in a communication domain. For example, a GPU card can belong to different communication domains, such as rank0 in communication domain 1, rank3 in communication domain 2, etc. Here, a GPU can be GPU card 1 in a server, or rank0 in a communication domain, etc.

[0084] Step S202: Create shared memory based on the identifier of the communication domain, calculate the offset of the memory space based on the communication number of the receiving GPU and the sending GPU to determine the starting address for allocating memory space for each GPU, the receiving GPU stores traffic receiving information, and the sending GPU stores traffic sending information.

[0085] A corresponding shared memory is created based on the communication domain ID to store communication traffic statistics. This shared memory is allocated to each GPU, with each GPU occupying a certain amount of memory space. The length of this memory space can be a preset length in bytes, such as 16 bytes, used to store traffic reception and transmission information. For example, the first 8 bytes store transmission information, and the last 8 bytes store reception information. The sending and receiving GPUs have different communication numbers; for example, the receiving GPU's communication number is rank0, and the sending GPU's is rank1. Based on the starting address of the shared memory, the length of the memory space, and the communication numbers of the receiving and sending GPUs, the offset of each GPU's memory space can be calculated to determine the starting address for allocating memory space to each GPU. For example, if the shared memory is divided into 32 bytes according to the communication number, the memory space for the receiving GPU (rank0) is the first 16 bytes, and the memory space for the sending GPU (rank1) is the last 16 bytes.

[0086] The receiving GPU generates traffic reception information and can store it; the sending GPU generates traffic transmission information and stores it.

[0087] Step S203: Use memory mapping to map the address of the traffic receiving information of each GPU to the process address space, and store the mapped address of the traffic receiving information in the receiving structure of the receiving GPU; use memory mapping to map the address of the traffic sending information of each GPU to the process address space, and store the mapped address of the traffic sending information in the sending structure of the sending GPU.

[0088] This embodiment adds pointers to the first structure of the GPU for tracking communication traffic. For example, a `recv` pointer is added to the receiver GPU's `RecvComm` structure, and a `send` pointer is added to the sender GPU's `SendComm` structure. Adding pointers to the first structure does not affect the compatibility of existing API interfaces.

[0089] Memory mapping methods such as mmap can be used to map the address of the traffic receiving information to the process address space, and the mapped address of the traffic receiving information is stored in a pointer to the receiving structure RecvComm of the receiving GPU. Similarly, memory mapping methods such as mmap can be used to map the address of the traffic sending information to the process address space, and the mapped address of the traffic sending information is stored in a pointer to the sending structure SendComm of the sending GPU.

[0090] Steps S202 and S203 can be implemented by constructing an NCCL API interface. Parameters include communication domain identifiers and GPU communication numbers, facilitating the allocation of shared memory based on the communication domain identifier, memory space allocation based on the communication number, and storage of communication traffic statistics in pointers to the send and receive structures. These operations are completed after the communication connection is established but before the GPU accesses data, ensuring minimal performance overhead. The API calls in steps S201, S202, and S203 can be pre-configured to ensure the normal execution of API calls.

[0091] Step S204: When the receiving GPU accesses the data of the sending GPU, it initiates a receiving request and passes a receiving structure, and updates the receiving pointer in the second structure according to the pointer of the receiving structure; the sending GPU initiates a sending request and passes a sending structure, and updates the sending pointer in the second structure according to the pointer of the sending structure; the traffic receiving information and / or traffic sending information of each GPU are determined according to the receiving pointer and / or sending pointer of the second structure.

[0092] In addition to adding pointers to the first structure of the GPU, this embodiment also adds pointers to the second structure used for communication, such as the request structure, and adds receive pointers and send pointers to the second structure.

[0093] When the receiving GPU accesses data from the sending GPU, it initiates a receive request and passes a receive structure RecvComm, along with a second structure as parameters. The receive pointer in the second structure is updated based on the recv pointer of the receive structure RecvComm. Conversely, the sending GPU initiates a send request and passes a send structure SendComm. The send pointer in the second structure is updated based on the send pointer of the send structure SendComm. Since both the receive and send pointers are updated in the second structure, the traffic reception and transmission information of each GPU in the communication domain can be determined based on the receive and send pointers in the second structure.

[0094] For specific procedures, please refer to the following: Figure 3aAs shown, the receiving GPU is represented by communication number 0, such as rank_0, and the sending GPU is represented by communication number 1, such as rank_1. During initialization, communication number 0 and communication number 1 each construct their own ncclComm structure. Communication number 0 uses the IbListen interface to listen for connection requests, while communication number 1 uses the IbConnect interface to initiate a connection request and, upon completion, creates a SendComm structure and saves it to the ncclComm structure of communication number 1. Communication number 0 uses the IbAccept interface to receive and establish a connection (communication connection), and upon completion, creates a RecvComm structure and saves it to the ncclComm structure of communication number 0. The `IbSetRank` interface is used to execute steps S202 and S203. It creates shared memory with the respective communication number `rank_0` or `rank_1` and the communication domain identifier `task_id` as parameters. Communication number 0 stores traffic reception information; after mmap memory mapping, the address of the `recv` pointer is saved to `RecvComm`. Communication number 1 stores traffic transmission information; after mmap memory mapping, the address of the `send` pointer is saved to `SendComm`. When data access occurs, communication number 0 uses the `IbIrecv` interface, with `RecvComm` and the second structure `IbRequest` as parameters, to update the `recv` pointer in `RecvComm` to the second structure `IbRequest`. Communication number 1 uses the `IbIsend` interface, with `SendComm` and the second structure `IbRequest` as parameters, to update the `send` pointer in `SendComm` to the second structure `IbRequest`. The `IbTest` interface can be used to detect whether the send / receive task is completed, and can update the traffic statistics based on the pointer of the second structure `IbRequest`. The IbIrecv and IbIsend interfaces can update the receive or send pointer in the second structure IbRequest. After the data sending and receiving task is completed in the IbTest interface, the traffic can be monitored based on the receive and send pointers in IbRequest.

[0095] Furthermore, when monitoring traffic, the traffic reception information and traffic transmission information of different GPUs can be determined based on the receive pointer and send pointer of the second structure, that is, the traffic monitoring situation of different GPUs included in the statistical communication domain can be statistically analyzed.

[0096] Alternatively, based on the receive and send pointers of the second structure, the total traffic received and sent information of multiple GPUs under the same distributed node can be statistically analyzed. For example... Figure 3bAs shown, in the distributed cluster, node 0 includes 4 GPUs with communication numbers rank0, rank1, rank2, and rank3, and node 1 includes 4 GPUs with communication numbers rank4, rank5, rank6, and rank7. Nodes 0 and 1 belong to the same communication domain and execute the same task 0. The data communication direction is as follows: Figure 3b As shown by the arrows, through traffic monitoring, it can be determined that node 0 receives 189.4GB of traffic from node 1 in rank 0 and sends 562.45GB of traffic to node 1 in rank 3. Node 1 receives 562.45GB of traffic from node 0 in rank 4 and sends 189.4GB of traffic to node 0 in rank 7. This achieves the goal of statistically analyzing the traffic monitoring of each node at the task granularity.

[0097] Alternatively, based on the receive and transmit pointers of different second structures, it is also possible to statistically analyze the traffic reception and transmission information belonging to different communication domains within the same GPU. For example... Figure 3c As shown, a GPU has different communication numbers in different communication domains, namely communication number 3, communication number 5, and communication number 6. Among them, communication number 3 executes task 1, communication number 5 executes task 2, and communication number 6 executes task 3. According to the receive pointer and send pointer of different second structures, it can be statistically determined that when communication number 3 executes task 1, the received traffic is 0GB and the send traffic is 38.56GB; when communication number 5 executes task 2, the received traffic is 77.56GB and the send traffic is 0GB; when communication number 6 executes task 3, the received traffic is 45.8GB and the send traffic is 0GB, etc.

[0098] The above traffic monitoring can be configured according to the implementation situation, and statistics can be performed in different units such as communication domain (tasks executed in the communication domain), GPU, and node to achieve multi-dimensional traffic monitoring.

[0099] According to the traffic monitoring method provided in this application, shared memory is created for the sending GPU and the receiving GPU, and memory space is allocated to store communication traffic statistics. The addresses storing the communication traffic statistics are mapped and stored in pointers to the receiving and sending structures of the sending and receiving GPUs, respectively. When the sending and receiving GPUs access data, the pointers in the receiving and sending structures are updated to the corresponding sending and receiving pointers in the second structure. This only adds pointers to each structure, does not affect the compatibility of various APIs, and has low performance overhead. Based on the sending and receiving pointers in the second structure, traffic monitoring can be performed from different dimensions such as communication domain granularity, node granularity, and GPU granularity, enabling traffic monitoring at different dimensions, including the same communication domain, GPUs belonging to different communication domains, and multiple GPUs contained under a node. This facilitates traffic monitoring during multi-task parallelism, faster identification of execution bottlenecks, and convenient timely problem detection and resolution.

[0100] Figure 4 A schematic diagram of the structure of a flow monitoring device according to an embodiment of this application is shown. Figure 4 As shown, the device includes:

[0101] Establishment module 410 is suitable for establishing communication connections based on multiple GPUs in a communication domain;

[0102] The shared memory module 420 is adapted to create shared memory based on the identifier of the communication domain and allocate corresponding memory space for multiple GPUs to store communication traffic statistics of each GPU; the communication traffic statistics include traffic reception information and / or traffic transmission information;

[0103] Address storage module 430 is adapted to use memory mapping to map the address of communication traffic statistics information to the process address space, and store the mapped address in the first structure of the GPU; wherein, the first structure is initialized when the communication connection is established, and the first structure points to the mapped address with a pointer;

[0104] The traffic acquisition module 440 is adapted to store communication traffic statistics information of the GPU to the address of the corresponding memory space when multiple GPUs access data, and update the pointer of the second structure of the communication request using the pointer of the first structure, so as to obtain the traffic reception information and / or traffic transmission information of each GPU according to the second structure.

[0105] Optionally, the multiple GPUs include a sending GPU and a receiving GPU; the first structure includes a receiving structure and / or a sending structure;

[0106] The connection module 410 is further adapted to:

[0107] The receiving GPU listens for connection requests. When it receives a connection request from the sending GPU, the receiving GPU accepts the connection request and establishes a communication connection. The receiving GPU creates a receive structure, and the sending GPU creates a send structure. The sending GPU and the receiving GPU belong to the same communication domain.

[0108] Optionally, the length of the memory space is a preset byte length; the communication numbers of the sending GPU and the receiving GPU are different;

[0109] Shared memory module 420 is further adapted to:

[0110] Create shared memory based on the identifier of the communication domain;

[0111] The offset of the memory space is calculated based on the communication number of the receiving GPU and the sending GPU to determine the starting address for allocating memory space for each GPU.

[0112] The receiving GPU stores the received traffic information, and the sending GPU stores the sent traffic information.

[0113] Optionally, the address storage module 430 is further adapted to:

[0114] Memory mapping is used to map the addresses of traffic receiving information of each GPU to the process address space, and the mapped addresses of traffic receiving information are stored in the receiving structure of the receiving GPU.

[0115] Memory mapping is used to map the addresses of traffic transmission information of each GPU to the process address space, and the mapped addresses of traffic transmission information are stored in the transmission structure of the sending GPU.

[0116] Optionally, the device further includes a pointer addition module 450, adapted to add a receive pointer and a transmit pointer to the second structure.

[0117] Optionally, the traffic acquisition module 440 is further adapted to:

[0118] When the receiving GPU accesses the data of the sending GPU, it initiates a receive request and passes a receive structure, and updates the receive pointer in the second structure according to the pointer of the receive structure; the sending GPU initiates a send request and passes a send structure, and updates the send pointer in the second structure according to the pointer of the send structure.

[0119] The traffic reception information and / or traffic transmission information of each GPU are determined based on the receive pointer and / or send pointer of the second structure.

[0120] Optionally, the traffic acquisition module 440 is further adapted to:

[0121] The receive and / or send information of different GPUs in the communication domain is determined based on the receive pointer and / or send pointer of the second structure;

[0122] Alternatively, based on the receive pointer and / or send pointer of the second structure, the total traffic received information and / or total traffic sent information of multiple GPUs under the same distributed node can be counted.

[0123] Alternatively, based on the receive pointers and / or send pointers of different second structures, statistical analysis can be performed on traffic reception and / or traffic transmission information belonging to different communication domains within the same GPU.

[0124] Optionally, the device further includes: a configuration module 460, adapted to pre-configure communication links of multiple GPUs in the communication domain so that the multiple GPUs can execute according to the communication links; the communication links include a first structure that establishes communication connections, creates shared memory, allocates memory space, stores communication traffic statistics, and maps addresses to the GPUs.

[0125] The descriptions of the above modules refer to the corresponding descriptions in the method embodiments, and will not be repeated here.

[0126] According to the traffic monitoring device provided in this application, based on the shared memory storage of GPU communication traffic statistics in the communication domain, the address of the communication traffic statistics is mapped and stored in the pointer of the first structure. When accessing data, the first structure is used to update the second structure. Based on the second structure, traffic monitoring of the communication domain can be realized from the granularity of the communication domain, realizing traffic monitoring when multiple tasks are parallel, and further helping to monitor and evaluate the network performance and health of distributed clusters used for large model training, etc.

[0127] This application also provides a non-volatile computer storage medium storing at least one executable instruction that can execute the traffic monitoring method in any of the above method embodiments.

[0128] Figure 5 The diagram shows a structural schematic of a computing device according to an embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the computing device.

[0129] like Figure 5 As shown, the computing device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0130] in:

[0131] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508.

[0132] Communication interface 504 is used to communicate with other network elements such as clients or other servers.

[0133] The processor 502 is used to execute program 510, which can specifically execute the relevant steps in the above-described traffic monitoring method embodiment.

[0134] Specifically, program 510 may include program code that includes computer operation instructions.

[0135] The processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement this application. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0136] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0137] Specifically, program 510 can be used to cause processor 502 to execute the traffic monitoring method in any of the above method embodiments. The specific implementation of each step in program 510 can be found in the corresponding descriptions of the steps and units in the above traffic monitoring embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0138] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing preferred embodiments of this application.

[0139] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0140] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0141] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0142] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0143] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0144] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A traffic monitoring method, comprising: Establish communication connections based on multiple GPUs in the communication domain; Shared memory is created based on the identifier of the communication domain, and corresponding memory space is allocated to the multiple GPUs to store communication traffic statistics of each GPU; The communication traffic statistics include traffic reception information and / or traffic transmission information; The address of communication traffic statistics is mapped to the process address space using memory mapping, and the mapped address is stored in the first structure of the GPU; wherein, the first structure is initialized when the communication connection is established, and the first structure is pointed to by a pointer to the mapped address; When multiple GPUs access data, the GPU stores the communication traffic statistics information to the address of the corresponding memory space, and updates the pointer of the second structure of the communication request using the pointer of the first structure, so as to obtain the traffic reception information and / or traffic transmission information of each GPU according to the second structure.

2. The method according to claim 1, wherein, The plurality of GPUs includes a sending GPU and a receiving GPU; the first structure includes a receiving structure and / or a sending structure; The establishment of a communication connection based on multiple GPUs in the communication domain further includes: The receiving GPU listens for connection requests. When it receives a connection request initiated by the sending GPU, the receiving GPU accepts the connection request and establishes a communication connection. The receiving GPU creates a receiving structure, and the sending GPU creates a sending structure. The sending GPU and the receiving GPU belong to the same communication domain.

3. The method according to claim 2, wherein, The length of the memory space is a preset byte length; the communication numbers of the sending GPU and the receiving GPU are different; The step of creating shared memory based on the identifier of the communication domain and allocating corresponding memory space for the multiple GPUs to store communication traffic statistics of each GPU further includes: Create shared memory based on the identifier of the communication domain; The offset of the memory space is calculated based on the communication number of the receiving GPU and the sending GPU to determine the starting address for allocating memory space for each GPU. The receiving GPU stores traffic reception information, and the sending GPU stores traffic transmission information.

4. The method according to claim 3, wherein, The step of mapping the address of communication traffic statistics information to the process address space using memory mapping, and storing the mapped address in the first structure of the GPU, further includes: The addresses of traffic receiving information of each GPU are mapped to the process address space using memory mapping, and the mapped addresses of traffic receiving information are stored in the receiving structure of the receiving GPU. The addresses of traffic transmission information of each GPU are mapped to the process address space using memory mapping, and the mapped addresses of traffic transmission information are stored in the transmission structure of the sending GPU.

5. The method according to any one of claims 2-4, wherein, The method further includes: Add a receive pointer and a send pointer to the second structure.

6. The method according to claim 5, wherein, When multiple GPUs access data, the GPU stores the communication traffic statistics information to the corresponding memory address, and updates the pointer of the second structure of the communication request using the pointer of the first structure, so as to obtain the traffic reception information and / or traffic transmission information of each GPU according to the second structure, further includes: When the receiving GPU accesses the data of the sending GPU, it initiates a receiving request and passes the receiving structure, and updates the receiving pointer in the second structure according to the pointer of the receiving structure; the sending GPU initiates a sending request and passes the sending structure, and updates the sending pointer in the second structure according to the pointer of the sending structure. The traffic reception information and / or traffic transmission information of each GPU are determined based on the receive pointer and / or send pointer of the second structure.

7. The method according to claim 6, wherein, The step of obtaining the traffic reception information and / or traffic transmission information of each GPU according to the second structure further includes: The receive and / or send information of different GPUs in the communication domain is determined based on the receive pointer and / or send pointer of the second structure; Alternatively, based on the receive pointer and / or send pointer of the second structure, the total traffic received information and / or total traffic sent information of multiple GPUs under the same distributed node can be counted. Alternatively, based on the receive pointers and / or send pointers of different second structures, statistical analysis can be performed on traffic reception and / or traffic transmission information belonging to different communication domains within the same GPU.

8. The method according to any one of claims 1-7, wherein, The method further includes: A communication link for multiple GPUs in a communication domain is pre-configured so that the multiple GPUs can execute according to the communication link; the communication link includes a first structure for establishing a communication connection, creating shared memory, allocating memory space, storing communication traffic statistics, and mapping addresses to be stored in the GPU.

9. A flow monitoring device, comprising: A connection module is established, suitable for establishing communication connections between multiple GPUs in a communication domain; A shared memory module is adapted to create shared memory based on the identifier of the communication domain and allocate corresponding memory space for the multiple GPUs to store communication traffic statistics of each GPU; The communication traffic statistics include traffic reception information and / or traffic transmission information; The address storage module is adapted to use memory mapping to map the address of communication traffic statistics information to the process address space, and store the mapped address in the first structure of the GPU; wherein, the first structure is initialized when the communication connection is established, and the first structure points to the mapped address with a pointer; The traffic acquisition module is adapted to allow the GPU to store the communication traffic statistics information to the address of the corresponding memory space when multiple GPUs access data, and to update the pointer of the second structure of the communication request using the pointer of the first structure, so as to obtain the traffic reception information and / or traffic transmission information of each GPU according to the second structure.

10. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the traffic monitoring method as described in any one of claims 1-8.

11. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the traffic monitoring method as described in any one of claims 1-8.

Images