Small-scale data AllGather communication method based on NCCL in stand-alone environment

Abstract

The invention provides a small-scale data AllGather communication method based on NCCL in a stand-alone environment, and belongs to the field of high-performance communication. The invention aims to solve the problem of low efficiency of the NCCL when processing small-scale data, and aims to realize the optimal communication effect under different data volume scenes. According to the main technical scheme, the method comprises the steps of data length judgment, shared memory initialization, data preparation and copying, a notification mechanism implemented by a ring topological structure and data collection. According to the technical scheme, the original AllGather communication method of the NCCL and the newly proposed small-scale data AllGather communication method are combined, the communication time delay is remarkably reduced, and the data exchange efficiency between the nodes is improved. The method has the main technical effects that the data transmission efficiency is improved, the shared memory is utilized and optimized, the data integrity is ensured, the automatic adaptive capacity is enhanced, and the compatibility with the NCCL native AllGather communication method is realized.

Description

Technical Field

[0001] The present invention belongs to the field of high-performance computer communication, including high-performance computing, cluster communication, etc., and specifically relates to a method for All_Gather communication of small-scale data based on NCCL in a single-machine environment. Background Art

[0002] NCCL (NVIDIA Collective Communications Library) is a high-performance cluster communication library developed by NVIDIA. All_Gather is a core communication mode in NCCL. Each node sends its own data to all other nodes and receives data from other nodes, and merges the received data with the local data. Compared with other communication modes, All_Gather can complete data collection and distribution in a single operation, reducing network overhead. In large-scale distributed systems, the efficiency of All_Gather is crucial for the overall training performance.

[0003] Small-scale data plays an important role in various application scenarios, especially in cases where low latency, high efficiency, or resource constraints are required, such as real-time communication systems, Internet of Things (IoT) device communication, etc. Efficient small-scale data transmission can help optimize network performance, reduce bandwidth usage, and adapt to various specific application requirements. However, for small-scale data, the advantages of the All_Gather method of NCCL are not obvious because the implementation of NCCL is optimized for multi-GPU and cross-node communication. Therefore, there is still much room for improvement in the existing All_Gather methods when dealing with small-scale data.

[0004] Aiming at the problems of relatively long latency and low communication efficiency when the existing All_Gather method of NCCL processes small-scale data, the present invention proposes a new algorithm, which optimizes the All_Gather communication method of small-scale data in a single-machine environment based on the existing NCCL. Summary of the Invention

[0005] The purpose of the present invention is to propose a brand-new algorithm for processing small-scale data according to the characteristics of the All_Gather method in the original NCCL, and use an innovative hybrid strategy design to significantly reduce the communication latency between small-scale data, improve the data exchange efficiency between nodes, and achieve the best communication effect.

[0006] The present invention provides a method for All_Gather communication of small-scale data based on NCCL in a single-machine environment, and the specific steps are as follows:

[0007] S1 Detect the data length of the data to be sent and set a threshold m;

[0008] When the data length is greater than m, use the All_Gather communication method of NCCL for data transmission;

[0009] When the data length is less than or equal to m, use the All_Gather communication method for small-scale data for data transmission.

[0010] In step S2, the All_Gather communication method of NCCL includes:

[0011] (2.1) Initialize the NCCL communication environment, initialize the communication cluster through the ncclCommInitAll function to ensure the establishment of communication channels between all nodes;

[0012] (2.2) Allocate device memory, allocate an independent device memory area for each node, and use the memory management interface provided by NCCL for data storage and transmission preparation;

[0013] (2.3) Data preparation and copying, each node copies the data to be sent to the specified location in the device memory to ensure that parallel transmission can be performed after the data is prepared;

[0014] (2.4) Start the All_Gather operation, call the ncclAllGather function to perform the data collection operation, and all nodes transmit data in parallel to other nodes to ensure that each node can obtain the data of other nodes;

[0015] (2.5) Synchronization and completion, coordinate the synchronization of each node through the ncclGroupStart and ncclGroupEnd functions to ensure that the All_Gather operation is completed consistently among all nodes;

[0016] (2.6) Resource release, after the operation is completed, destroy the communication environment and release relevant resources through ncclCommDestroy to avoid memory leakage.

[0017] In step S3, the All_Gather communication method for small-scale data includes:

[0018] (3.1) Initialize the shared memory area, design the shared memory naming rule, and allocate and initialize the shared memory space;

[0019] (3.2) Data preparation and copying, copy the data to be sent to the predetermined location in the shared memory, and the positions of the data of each node in the shared memory are continuous and unique;

[0020] (3.3) Build a notification mechanism. Each node continuously polls the notifyRank flag in the shared memory. When the notifyRank value is equal to the Rank value of the current node, update the notifyRank to the Rank value of the predecessor node; otherwise, continue polling.

[0021] (3.4) Data collection and complete data reading. Each node polls the notifyRank value. When the notifyRank value is equal to the Rank value of the first node, read the complete All_Gather data.

[0022] In step (3.1), initialize the shared memory area, including:

[0023] (3.1.1) Set the naming rule for the shared memory, and define the name of the shared memory area as the unique / dev / shm / all_gather / <commhash>, where all nodes within the same cluster <commhash>The same. This naming method ensures that all nodes within the same cluster have the same shared memory area name and can jointly access the same physical memory area;

[0024] (3.1.2) Initialize a shared memory area with a size of m * nRanks + 4 bytes, where nRanks represents the number of nodes in the cluster; m is an adjustable threshold, that is, the upper limit of the data stored by each node is m, and the maximum space required for all nodes to store is m * nRanks bytes; the last 4 bytes are used to store an int - type notifyRank flag. Using a fixed position, all nodes can quickly locate and obtain the correct notifyRank value. This flag is used to control the communication process, and the initial value is set to the Rank value of the first node.

[0025] In step (3.2), data preparation and copying include: when the communication cluster starts the All_Gather operation, all nodes copy the data to be sent to a specific location in the shared memory. There are nRanks nodes in total. The i - th (i = 0, 1...n) node is denoted as rank_i, and the length of the data to be sent by the node is denoted as count. Then the offset address offset for the node to copy in the shared memory is offset = count * i. All nodes complete the data transfer concurrently.

[0026] In step (3.3), construct a notification mechanism, including:

[0027] (3.3.1) Use the ring topology structure generated during NCCL initialization. Each node has a predecessor node preRank and a successor node nextRank;

[0028] (3.3.2) Each node continuously polls the notifyRank value in the shared memory; when the notifyRank value is equal to the current node's Rank value, immediately update the notifyRank value in the shared memory to the value of the predecessor node preRank of the current node in the ring topology structure; otherwise, continue polling.

[0029] In step (3.4), data collection and complete data reading include: each node continuously polls the notifyRank value in the shared memory; when the notifyRank value is equal to the Rank value of the first node, it indicates that the data copying process of all nodes has ended. Each node immediately reads the complete All_Gather data from the shared memory, that is, count * nRanks data starting from the starting position; otherwise, continue polling.

[0030] On the other hand, the present invention also provides a computer device, including a processor, a memory, a storage, and a network interface, characterized in that: the processor executes the small-scale data All_Gather communication method based on NCCL in a single-machine environment according to any one of steps 1 to 3, supports multi-threaded processing, and optimizes the parallel computing performance; the memory is used to run parallel computing programs and store temporary data, including a shared memory part for realizing high-speed data exchange across nodes; the storage is used to save the operating system, the parallel computing framework, and relevant configuration files; the network interface is used to connect multiple computer devices to form a distributed computing cluster.

[0031] The hybrid All_Gather communication scheme proposed by the present invention realizes the best communication effect in different data volume scenarios by cleverly combining the NCCL native method and the newly proposed small-scale data optimization algorithm. The main functions of this dual-strategy design include: automatically adapting to the communication requirements of different data sizes, maintaining high efficiency in large data volumes, and at the same time significantly improving the communication efficiency of small data volumes. Its core advantages are reflected in the following aspects:

[0032] First of all, this design has strong compatibility and can be seamlessly integrated into the existing NCCL framework without modifying the original code structure, reducing the implementation cost and risk; the adaptive mechanism enables the system to automatically select the most suitable communication method according to the actual data size without manual intervention, greatly improving the convenience and flexibility of use.

[0033] Secondly, efficient use of shared memory. Cleverly use the only <commhash>Naming was carried out to create a communication area based on shared memory between GPUs. Without network communication, all nodes can directly access the same physical memory data, significantly reducing communication latency.

[0034] Furthermore, an intelligent synchronization mechanism. The value of notifyRank is initialized to the Rank value of the first node. After the first node finishes copying, notifyRank is modified to the value of preRank, the predecessor node in the ring topology. Each node performs this operation in turn until it returns to the first node again, at which point it proves that the copy process of all nodes has been completed. The design of notifyRank is used to determine whether the copy process of a specific node's data to be sent to shared memory is completed during communication. Through the ring topology and the update mechanism of notifyRank, a lock-free synchronization barrier function is achieved. This ensures that all nodes can read complete data and avoids data inconsistency.

[0035] Finally, parallel data copying. Each node copies data of the same size to shared memory almost simultaneously, making full use of the parallel processing capabilities of multiple nodes. This reduces the total time of data transmission and improves the overall communication efficiency.

[0036] In summary, the present invention proposes a new algorithm to make up for the deficiencies of the All_Gather communication method of NCCL when dealing with small-scale data. Using the ideas of shared memory and CAS (Compare and Swap), and the design of notifyRank, it ensures the simplicity, low latency, and high efficiency of the algorithm.

[0037] The present invention proposes a small-scale data All_Gather communication method based on NCCL in a single-machine environment, which is a hybrid All_Gather communication scheme. It solves the problems existing in NCCL when dealing with small-scale data and provides a more efficient, flexible, and compatible communication solution for distributed computing. By optimizing data transmission efficiency, improving shared memory utilization, ensuring data integrity, enhancing automatic adaptation ability, and maintaining compatibility, the following key technical effects are achieved:

[0038] Improved data transmission efficiency: Each node copies the data to be sent to a specific location in shared memory and copies the complete data to all nodes through shared memory. This design enables all nodes to complete data Gather with only two data copies, greatly reducing the time overhead of data transmission and significantly improving communication efficiency.

[0039] Shared Memory Utilization Optimization: The present invention ingeniously utilizes shared memory and a ring topology to achieve fast and low-latency data exchange. This method is particularly suitable for scenarios that require frequent processing of a large number of small-scale data, such as real-time communication systems, IoT device communication, etc.

[0040] Data Integrity Assurance: The design of notifyRank in shared memory ensures that the Gather results copied by all nodes in the shared memory are complete, and without using time-consuming processing methods such as locks, it realizes the integrity confirmation of shared memory data and avoids potential race condition problems.

[0041] Enhanced Automatic Adaptability: By setting an adjustable threshold, the system can automatically select the most suitable communication method according to the data size. This not only improves communication efficiency but also greatly reduces the complexity of system configuration and maintenance, enabling the solution to flexibly adapt to various application scenarios with different data magnitudes.

[0042] Compatibility Maintenance: The present invention retains the native All_Gather communication method of NCCL and still uses the original NCCL solution for large-scale data. This compatibility design enables the proposed All_Gather communication method for small-scale data in the present invention to be seamlessly integrated into the existing distributed computing environment without major modifications. Brief Description of the Drawings

[0043] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0044] Figure 1 It is a schematic diagram of the overall design process;

[0045] Figure 2 It is a schematic diagram of the All_Gather communication process for small-scale data;

[0046] Figure 3 It is a schematic diagram of the shared memory area design;

[0047] Figure 4 It is a schematic diagram of the data distribution in the shared memory area;

[0048] Figure 5 It is an experimental result diagram of the original NCCL All_Gather communication method;

[0049] Figure 6 It is an experimental result diagram after optimization for small-scale data;

[0050] Figure 7 It is a schematic diagram for comparing the communication delay of All_Gather;

[0051] Figure 8 It is a schematic diagram of the computer device structure. Specific implementation manners

[0052] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this invention belongs. The terms used in the description of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0054] In this embodiment, in a single machine with multiple cards environment, the hybrid All_Gather communication scheme proposed by the present invention is used to optimize the efficiency of the original All_Gather communication in NCCL.

[0055] Embodiment 1

[0056] As Figure 1 shown, this embodiment adopts a hybrid All_Gather communication scheme, including the following steps:

[0057] S1. Detect the data length of the data to be sent, and set the threshold to 16 KB.

[0058] S2. When the data length is greater than 16 KB, use the All_Gather communication method of NCCL for data transmission, including the following steps:

[0059] S2.1. Initialize the NCCL communication environment, initialize the communication cluster through the ncclCommInitAll function to ensure the establishment of communication channels between all nodes;

[0060] S2.2. Allocate device memory, allocate an independent device memory area for each node, and use the memory management interface provided by NCCL for data storage and transmission preparation;

[0061] S2.3. Data preparation and copying, each node copies the data to be sent to the specified position in the device memory to ensure that parallel transmission can be performed after the data is prepared;

[0062] S2.4. Start the All_Gather operation, call the ncclAllGather function to perform the data collection operation, and all nodes transfer data to other nodes in parallel to ensure that each node can obtain the data of other nodes;

[0063] S2.5. Synchronization and completion. Coordinate the synchronization of each node through the ncclGroupStart and ncclGroupEnd functions to ensure that the All_Gather operation is completed consistently among all nodes;

[0064] S2.6. Resource release. After the operation is completed, destroy the communication environment and release relevant resources through ncclCommDestroy to avoid memory leakage.

[0065] S3. When the data length is less than or equal to 16KB, as Figure 2 shown, use the small-scale data All_Gather communication method for data transmission, including the following steps:

[0066] S3.1. Initialize the shared memory area.

[0067] S3.1.1. Shared memory naming rule. Define the name of the shared memory area as the unique / dev / shm / all_gather / <commhash>, where all nodes within the same cluster <commhash>The same. This naming method ensures that all nodes within the same cluster have the same shared memory area name and can jointly access the same physical memory area.

[0068] S3.1.2. Shared Memory Area Design. The specific design is as follows Figure 3 shown.

[0069] S3.1.2.1. Initialize a shared memory area with a size of 16K * nRanks + 4 bytes, where nRanks represents the number of nodes in the cluster; the adjustable threshold is set to 16KB, that is, the upper limit of the data volume that each node needs to store is 16KB, and the maximum space required for all nodes to store is 16K * nRanks bytes; the last 4 bytes are used to store an int - type flag, so the initialized shared memory area size is 16K * nRanks + 4 bytes.

[0070] S3.1.2.2. Use an int - type notifyRank flag in the shared memory to control the communication process. The initial value is set to the Rank of the first node. This flag not only simplifies the communication process but also provides a security mechanism to avoid data competition.

[0071] S3.1.3. Memory Layout.

[0072] S3.1.3.1. The last 4 bytes are used to store the notifyRank value. By having a fixed position, all nodes can quickly locate and obtain the correct notifyRank value.

[0073] S3.1.3.2. The remaining space is sequentially allocated to each node, and each node occupies 16KB of space. This allocation method not only ensures that each node has enough space to store data but also maintains the space utilization efficiency. The 16KB space of each node can be flexibly used to store data to be sent, achieving efficient data buffering.

[0074] S3.2. Data Preparation and Copying.

[0075] S3.2.1. Unified Data Length. The length of the data to be sent by each node is denoted as count.

[0076] S3.2.2. Copy Data to Shared Memory.

[0077] S3.2.2.1. When the communication cluster starts the All_Gather operation, all nodes copy the data to be sent to a specific location in the shared memory. This step is a key link in the entire communication process, and all nodes complete data transmission concurrently, efficiently, and accurately.

[0078] S3.2.2.2. There are nRanks nodes in total. The i-th (i = 0, 1... n) node is denoted as rank_i, and the offset address offset for copying this node in the shared memory is offset = count * i. This design ensures that the data of each node is continuous and unique in the shared memory.

[0079] S3.2.3. Schematic diagram of data distribution. The nodes in the shared memory area are arranged in sequence, and the data distribution design in the shared memory is as Figure 4 shown. This data layout not only ensures the correctness of the data but also improves the subsequent data reading and processing efficiency.

[0080] S3.3. Build a notification mechanism.

[0081] S3.3.1. Ring topology structure. Use the ring topology structure generated during NCCL initialization. Each node has a predecessor node preRank and a successor node nextRank.

[0082] S3.3.2. Notification polling.

[0083] S3.3.2.1. Each node continuously polls the notifyRank value in the shared memory. This design implements an efficient notification mechanism, enabling all nodes to promptly perceive changes in the communication status.

[0084] S3.3.2.2. When the notifyRank value is equal to the current node's Rank value, immediately update the notifyRank value in the shared memory to the value of the predecessor node preRank of this node in the ring topology structure.

[0085] S3.3.2.3. Otherwise, continue polling.

[0086] S3.4. Data collection to ensure complete data reading.

[0087] S3.4.1. Each node continuously polls the notifyRank value in the shared memory.

[0088] S3.4.2. When the notifyRank value is equal to the Rank value of the first node, each node immediately reads the complete All_Gather data from the shared memory, that is, count * nRanks data starting from the starting position. This step is crucial in the entire communication process. When the notifyRank value is equal to the Rank value of the first node, it indicates that the data copying process of all nodes has ended, ensuring that each node reads complete and correct data.

[0089] S3.4.3. Otherwise, continue polling.

[0090] The present invention uses the nccl-test, a testing tool provided by the official NCCL, to conduct experiments for specific hardware configurations to verify its effectiveness and performance advantages. The test environment is based on the Ubuntu 20.04 operating system, equipped with a high-performance Intel(R) Xeon(R) Gold 6348 CPU @ 2.60GHz processor, and three Tesla T4 GPUs are installed. This hardware configuration provides strong computing and communication capabilities for the experiment, enabling a full evaluation of the performance of NCCL. The focus of the test is on comparing the performance differences between using the original All_Gather communication method of NCCL and the All_Gather communication method optimized for small-scale data.

[0091] To comprehensively evaluate the performance of the two methods, based on the above hardware environment, a large number of experiments are conducted using the two methods respectively to collect accurate latency data.

[0092] Figure 5 The experimental result graph using the original All_Gather communication method of NCCL is shown. This graph provides the latency performance of the original method under various data scales, giving a global perspective, and based on this, it is convenient to compare and analyze the method proposed in the present invention.

[0093] Figure 6 The latency performance under various data scales after optimizing the All_Gather communication for small-scale data using the method proposed in the present invention is shown. Figure 5 and Figure 6 Using specific experimental data, objectively compare the performance differences between the two methods.

[0094] Figure 7 For Figure 5 and Figure 6 The experimental data in are summarized, and the latency differences between the two methods under different data scales are visually shown through a line graph. Figure 7 It can be visually seen from that after optimizing the All_Gather communication for small-scale data, the communication latency of small-scale data has been significantly reduced, and the upward trend is slower.

[0095] Figure 8 Disclosed is a computer device provided by the present invention, including a processor, a memory, a storage, and a network interface, characterized in that: the processor executes a small-scale data All_Gather communication method based on NCCL in a single-machine environment, supports multi-threaded processing, and optimizes parallel computing performance; the memory is used to run parallel computing programs and store temporary data, including a shared memory part for realizing high-speed data exchange across nodes; the storage is used to save the operating system, parallel computing framework, and related configuration files; the network interface is used to connect multiple computer devices to form a distributed computing cluster.

[0096] This research successfully breaks through the limitations of the original method in small-scale data processing by optimizing the All_Gather communication method in NCCL. Experimental results show that the performance of the optimized All_Gather communication method in processing small-scale data has been significantly improved, including a reduction in average latency and an improvement in overall stability. This optimization not only solves the performance bottleneck of the All_Gather method in NCCL for small-scale data but also improves the overall communication efficiency and reliability of the system.

[0097] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0098] The above-described embodiments only represent several implementation manners of the present invention. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the appended claims.< / commhash> < / commhash> < / commhash> < / commhash> < / commhash>

Claims

1. A small-scale data All_Gather communication method based on NCCL in a single-machine environment, characterized in that The method includes: S1. Detect the data length of the data to be sent and set a threshold m; S2. When the data length is greater than m, use the All_Gather communication method of NCCL for data transmission; S3. When the data length is less than or equal to m, use the All_Gather communication method for small-scale data for data transmission.

2. A small-scale data All_Gather communication method based on NCCL in a single-machine environment according to claim 1, characterized in that The All_Gather communication method of NCCL in step S2 includes: (2.1) Initialize the NCCL communication environment, initialize the communication cluster through the ncclCommInitAll function to ensure the establishment of communication channels between all nodes; (2.2) Allocate device memory, allocate an independent device memory area for each node, and use the memory management interface provided by NCCL for data storage and transmission preparation; (2.3) Data preparation and copying, each node copies the data to be sent to a specified location in the device memory to ensure that parallel transmission can be performed after the data is prepared; (2.4) Start the All_Gather operation, call the ncclAllGather function to perform the data collection operation, and all nodes transmit data to other nodes in parallel to ensure that each node can obtain the data of other nodes; (2.5) Synchronization and completion, coordinate the synchronization of each node through the ncclGroupStart and ncclGroupEnd functions to ensure that the All_Gather operation is completed consistently among all nodes; (2.6) Resource release, after the operation is completed, destroy the communication environment and release relevant resources through ncclCommDestroy to avoid memory leakage.

3. A small-scale data All_Gather communication method based on NCCL in a single-machine environment according to claim 1, characterized in that, The All_Gather communication method for small-scale data in step S3 includes: (3.1) Initialize the shared memory area, design the shared memory naming rule, and allocate and initialize the shared memory space; (3.2) Data preparation and copying, copy the data to be sent to a predetermined location in the shared memory, and the data of each node is continuous and unique in the shared memory; (3.3) Build a notification mechanism, each node continuously polls the notifyRank flag in the shared memory. When the notifyRank value is the Rank value of the current node, update notifyRank to the Rank value of the predecessor node, otherwise continue polling; (3.4) Data collection and complete data reading, each node polls the notifyRank value. When the notifyRank value is equal to the Rank value of the first node, read the complete All_Gather data.

4. A small-scale data All_Gather communication method based on NCCL in a single-machine environment according to claim 3, characterized in that, Initializing the shared memory area in step (3.1) includes: (3.1.1) Set the shared memory naming rule and define the name of the shared memory area as the unique / dev / shm / all_gather / <commhash>, where all nodes within the same cluster <commhash>Same. This naming method ensures that all nodes in the same cluster have the same shared memory area name and can jointly access the same physical memory area;< / commhash> < / commhash> (3.1.2) Initialize a shared memory area with a size of m * nRanks + 4 bytes, where nRanks represents the number of nodes in the cluster; m is an adjustable threshold, that is, the upper limit of the data volume stored by each node is m, and the maximum space required for all nodes to store is m * nRanks bytes; the last 4 bytes are used to store an int-type notifyRank flag. Using a fixed position, all nodes can quickly locate and obtain the correct notifyRank value. This flag is used to control the communication process, and the initial value is set to the Rank value of the first node.

5. The method for All_Gather communication of small-scale data based on NCCL in a single-machine environment according to claim 3, wherein In step (3.2), data preparation and copying include: when the communication cluster starts the All_Gather operation, all nodes copy the data to be sent to a specific location in the shared memory. There are nRanks nodes in total. The i-th (i = 0, 1... n) node is denoted as rank_i, and the length of the data to be sent by the node is denoted as count. Then the offset address offset for the node to copy in the shared memory is offset = count * i. All nodes complete the data transfer concurrently.

6. The method for All_Gather communication of small-scale data based on NCCL in a single-machine environment according to claim 3, wherein In step (3.3), constructing a notification mechanism includes: (3.3.1) Use the ring topology generated during NCCL initialization. Each node has a predecessor node preRank and a successor node nextRank; (3.3.2) Each node continuously polls the notifyRank value in the shared memory; when the notifyRank value is equal to the current node's Rank value, immediately update the notifyRank value in the shared memory to the value of the predecessor node preRank of the current node in the ring topology; otherwise, continue polling.

7. The method for All_Gather communication of small-scale data based on NCCL in a single-machine environment according to claim 3, wherein In step (3.4), data collection and complete data reading include: each node continuously polls the notifyRank value in the shared memory; when the notifyRank value is equal to the Rank value of the first node, it means that the data copying process of all nodes has ended. Each node immediately reads the complete All_Gather data from the shared memory, that is, count * nRanks data starting from the starting position; otherwise, continue polling.

8. A computer device, comprising a processor, a memory, a storage, and a network interface, characterized in that: The processor executes the small-scale data All_Gather communication method based on NCCL in a single-machine environment described in any one of claims 1 to 7, supports multi-threaded processing, and optimizes the parallel computing performance; the memory is used to run the parallel computing program and store temporary data, including a shared memory part for realizing high-speed data exchange across nodes; The memory is used to save the operating system, parallel computing framework, and related configuration files; The network interface is used to connect multiple computer devices to form a distributed computing cluster.

Images