Collective communication method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides a collection communication method and device, electronic equipment and storage medium, and relates to the technical field of high-performance parallel computing and distributed communication. The method comprises: traversing a plurality of transmission speeds in descending order, determining the current traversed transmission speed as a target transmission speed, and determining at least two ring communication paths formed by physical communication links between a plurality of computing devices based on the target transmission speed; wherein the plurality of transmission speeds respectively represent the expected communication rate of the ring communication paths to be constructed, each ring communication path comprises a plurality of computing devices, and any two ring communication paths do not share the same physical communication link; determining data shards corresponding to the at least two ring communication paths in the to-be-reduced data; and performing full-reduction communication on the data shards on the at least two ring communication paths to obtain a full-reduction result corresponding to the to-be-reduced data. The present scheme can reduce communication latency.

Description

Technical Field

[0001] This disclosure relates to the fields of high-performance parallel computing and distributed communication technology, and more specifically, to a collective communication method, apparatus, electronic device and storage medium. Background Technology

[0002] Collective communication is commonly used in high-performance parallel computing and distributed training for data synchronization, gradient aggregation, or computation result reduction. Full reduction communication is a typical example of collective communication operations, used to enable multiple computing devices to obtain the same global reduction result.

[0003] Existing full-reduction communication typically employs the Ring Allreduce scheme, which arranges multiple computing devices into a logical ring communication path in a predetermined order. Each computing device communicates point-to-point only with the preceding and subsequent computing devices in the logical ring communication path, and performs the reduction distribution phase and the full collection phase based on this logical ring communication path.

[0004] However, in this Ring Allreduce scheme, both the reduction distribution phase and the full collection phase require multiple rounds of communication along a single logical ring communication path. With an increased number of computing devices or a large amount of data to be reduced, this single logical ring communication path needs to continuously carry multiple rounds of transmission of the entire amount of data to be reduced, resulting in significant communication latency.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a communication method, apparatus, electronic device and storage medium that can reduce communication latency.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0008] According to a first aspect of the present disclosure, a collection communication method is provided, the method comprising: The system iterates through multiple transmission speeds in descending order, determines the currently traversed transmission speed as the target transmission speed, and, based on the target transmission speed, determines at least two ring communication paths formed by the physical communication links between multiple computing devices. The multiple transmission speeds represent the expected communication rate of the ring communication paths to be constructed. Each ring communication path includes multiple computing devices, and no two ring communication paths share the same physical communication link. Identify data fragments in the data to be standardized that correspond to at least two ring communication paths; On at least two ring communication paths, full reduction communication is performed on the data fragments to obtain the full reduction result corresponding to the data to be reduced.

[0009] The aggregate communication method in the example embodiments of this disclosure traverses multiple transmission speeds in descending order, determines the currently traversed transmission speed as the target transmission speed, and, based on the target transmission speed, determines at least two ring communication paths formed by physical communication links between multiple computing devices. The multiple transmission speeds represent the expected communication rate of the ring communication paths to be constructed. Each ring communication path includes multiple computing devices, and no two ring communication paths share the same physical communication link. Data fragments corresponding to the at least two ring communication paths are segmented from the data to be reduced, and full reduction communication is performed on the corresponding ring communication paths, so that the data to be reduced is carried by at least two ring communication paths respectively. Therefore, compared to the existing Ring Allreduce method where all data to be reduced is carried by a single logical ring communication path, this reduces the amount of data that a single ring communication path needs to transmit during full reduction communication, thereby reducing the total data transmission time of the full reduction communication process and lowering communication latency.

[0010] In some example embodiments of this disclosure, based on the foregoing scheme and the target transmission speed, at least two ring communication paths are determined to form the physical communication links between multiple computing devices, including: Identify the multiple computing devices participating in the full protocol communication, and the physical communication links between the multiple computing devices; Based on the target transmission speed and the physical communication links between multiple computing devices, at least two ring communication paths are constructed.

[0011] In this example embodiment, by identifying multiple computing devices participating in full protocol communication and the physical communication links between these devices, at least two ring communication paths are constructed based on the target transmission speed and the physical communication links. The construction of these ring communication paths is based on the actual existing physical communication links, ensuring that each constructed ring communication path is usable and deployable in the hardware environment, avoiding the generation of invalid paths that cannot be actually used or deployed. This ensures that full protocol communication is performed using at least two ring communication paths, guaranteeing the accuracy of communication and reducing communication latency.

[0012] In some example embodiments of this disclosure, based on the foregoing scheme, and based on the target transmission speed and the physical communication links between multiple computing devices, at least two ring communication paths are constructed, including: The path node device in the ring communication path to be built is selected hop by hop from multiple computing devices, and the bandwidth usage in this step is deducted from the remaining bandwidth of the physical communication link corresponding to the path node device. If the remaining bandwidth of the physical communication links corresponding to each path node device meets the bandwidth conditions corresponding to the target transmission speed, then a ring communication path is constructed based on the physical communication links corresponding to each path node device. Based on the physical communication links that are not used by the constructed ring communication paths, continue to construct the next ring communication path until at least two ring communication paths are obtained.

[0013] In this example embodiment, by selecting path node devices hop by hop and deducting the bandwidth occupancy of the corresponding physical communication link at each step, candidate ring communication paths can be dynamically constructed among multiple computing devices. This ensures that each path forms a closed ring communication path when the link bandwidth allows, and that the next ring communication path is constructed based on the remaining bandwidth not used by the constructed ring communication path. Multiple ring communication paths can be generated continuously, while ensuring that different ring communication paths do not repeatedly occupy physical communication links. This improves the link utilization and communication parallelism under the Full Mesh architecture, reduces the possibility of bandwidth conflicts, and provides efficient and reliable hardware resource guarantees for subsequent full protocol communication.

[0014] In some example embodiments of this disclosure, based on the foregoing scheme, the above method further includes: During the construction of the current ring communication path, if the remaining bandwidth of the physical communication link corresponding to the path node device of the current hop does not meet the bandwidth condition corresponding to the target transmission speed, the bandwidth occupancy of the physical communication link corresponding to the path node device that was deducted is restored, and the path node device of the current hop is reselected.

[0015] In this example embodiment, during the construction of the current ring communication path, when the remaining bandwidth of the physical communication link corresponding to the path node device of the current hop does not meet the bandwidth condition corresponding to the target transmission speed, the bandwidth occupancy of the physical communication link corresponding to the path node device is restored. This can restore the deducted bandwidth occupancy in a timely manner and reselect the path node device of the current hop, thereby ensuring that each ring communication path is successfully constructed under the premise of bandwidth availability. It can dynamically respond to the situation of insufficient link bandwidth, avoid bandwidth waste caused by node selection failure, improve the success rate of ring communication path construction, and at the same time ensure that different paths do not repeatedly occupy physical communication links.

[0016] In some example embodiments of this disclosure, based on the foregoing scheme, path node devices in the ring communication path to be constructed are selected hop-by-hop from multiple computing devices, including: From multiple computing devices, identify candidate node devices that have not joined the ring communication path to be built, and obtain the topology evaluation parameters of the candidate node devices; The evaluation results of candidate node devices are determined based on topology evaluation parameters; Based on the evaluation results of each candidate node device, the path node devices in the ring communication path to be constructed are selected hop by hop.

[0017] In this example embodiment, when selecting path node devices hop by hop, candidate node devices that have not been added to the ring communication path to be built are identified, and the topology evaluation parameters of the candidate node devices are obtained and the evaluation results are determined accordingly. Then, path node devices can be selected hop by hop according to the evaluation results, and computing devices with better topology conditions are selected into the ring communication path first. This can improve the overall communication performance of the finally constructed ring communication path and reduce the time for at least two ring communication paths to perform full protocol communication, that is, reduce communication latency.

[0018] In some example embodiments of this disclosure, based on the foregoing scheme, the above method further includes: Determine the target path between different computing devices from multiple computing devices; The path node devices in the ring communication path to be constructed are selected hop-by-hop from multiple computing devices, including: Based on the target path between different computing devices, the path node device in the ring communication path to be constructed is selected hop by hop from multiple computing devices.

[0019] In this example embodiment, the target path between different computing devices is first determined from multiple computing devices. Then, based on the target path, the path node device in the ring communication path to be constructed is selected hop by hop from multiple computing devices. This can directly reference the target path determined in the pre-computation stage, avoiding repeated calculation of the end-to-end path between different computing devices during the ring communication path search process. This allows for the faster construction of at least two ring communication paths, thereby reducing communication latency.

[0020] In some example embodiments of this disclosure, based on the foregoing scheme, determining the target path between different computing devices from multiple computing devices includes: For a first device and a second device among multiple computing devices, at least two candidate paths are determined between the first device and the second device, and the bottleneck bandwidth of each candidate path is determined; the bottleneck bandwidth is the minimum bandwidth of each physical communication link in the candidate path. If the bottleneck bandwidths of the candidate paths are different, the target path is determined from at least two candidate paths based on the bottleneck bandwidths. If the bottleneck bandwidth of each candidate path is the same, the hop count of each candidate path is determined, and the target path is determined from at least two candidate paths based on the hop count of each candidate path.

[0021] In this example embodiment, for at least two candidate paths between the first computing device and the second computing device, the bottleneck bandwidth of each candidate path is determined. If the bottleneck bandwidths are different, the target path is determined based on the bottleneck bandwidth. If the bottleneck bandwidths are the same, the target path is determined based on the number of hops. That is, the candidate path with the larger bottleneck bandwidth is given priority as the target path, so that the lower limit of the bandwidth of a single physical communication link in the final ring communication path is higher, reducing the risk of reducing the overall communication capability of the ring communication path due to individual low-bandwidth physical communication links. At the same time, when multiple candidate paths have the same bottleneck bandwidth, the target path with fewer hops is determined based on the number of hops. Therefore, the data passes through fewer intermediate computing devices during full protocol communication, further reducing communication latency.

[0022] In some example embodiments of this disclosure, based on the foregoing scheme, the above method further includes: If at least two ring communication paths cannot be constructed for the target transmission speed currently being traversed, then return to execute the steps of traversing multiple transmission speeds in descending order until at least two ring communication paths are constructed.

[0023] In this example embodiment, if at least two ring communication paths cannot be constructed for the currently traversed target transmission speed, the process returns to continue traversing multiple transmission speeds, determines the lower target transmission speed, and continues to construct ring communication paths based on the lower target transmission speed and physical communication links until at least two ring communication paths are constructed. This ensures good adaptability to physical communication links with different bandwidth resources when constructing ring communication paths, avoids the inability to construct ring communication paths due to excessively high transmission speeds, and improves the accuracy of constructing ring communication paths.

[0024] In some example embodiments of this disclosure, based on the foregoing scheme, the above method further includes: After constructing at least two ring communication paths, determine the total throughput of the at least two ring communication paths; With the total throughput remaining constant, the target transmission speed of at least two ring communication paths is reduced when the number of at least two ring communication paths is increased.

[0025] In this example embodiment, after constructing at least two ring communication paths, the number and speed of the ring communication paths can be dynamically adjusted by calculating the total throughput based on the target transmission speed and each ring path. When the number of ring communication paths is increased, the target transmission speed of the ring communication paths is reduced accordingly to keep the total throughput basically unchanged. This can improve the parallelism of the ring communication paths without increasing physical bandwidth consumption, hide the single-channel startup delay, optimize communication performance in small and medium message scenarios, and ensure stable total throughput. This provides an efficient and reliable hardware resource utilization strategy for multi-ring full-reference communication.

[0026] In some example embodiments of this disclosure, based on the foregoing scheme, determining data fragments in the data to be standardized that correspond to at least two ring communication paths includes: The data to be standardized is divided into multiple data fragments; Determine the mapping relationship between multiple data fragments and at least two ring communication paths; Based on the mapping relationship, data fragments corresponding to at least two ring communication paths are determined.

[0027] In this example embodiment, the data to be reduced is divided into multiple data fragments, the mapping relationship between the multiple data fragments and at least two ring communication paths is determined, and then the data fragments corresponding to each ring communication path are determined based on the mapping relationship. Then each ring communication path can obtain its own data fragments to perform full reduction communication, reducing communication latency.

[0028] In some example embodiments of this disclosure, based on the foregoing scheme, the data to be reduced is divided into multiple data fragments, including: Based on the number of at least two ring communication paths, the data to be standardized is divided into multiple data fragments.

[0029] In this example embodiment, the data to be rectified is divided into multiple data fragments based on the number of at least two ring communication paths. The number of data fragments matches the number of ring communication paths. Then, at least two ring communication paths can perform full rectification communication on their respective data fragments. This can reduce the amount of data that a single ring communication path needs to transmit during the full rectification communication process, thereby reducing the total data transmission time of the full rectification communication process and reducing communication latency.

[0030] In some example embodiments of this disclosure, based on the foregoing scheme, full protocol communication is performed on data fragments on at least two ring communication paths, including: On each ring communication path, the reduction distribution phase and the full collection phase of the full reduction communication are executed on the data fragments corresponding to the ring communication path; the reduction distribution phase and the full collection phase are executed in parallel or interleaved.

[0031] In this example embodiment, the reduction distribution phase and the full collection phase are executed on the corresponding data fragments of each ring communication path. The reduction distribution phase and the full collection phase on the ring communication path are executed in parallel or interleaved. That is, different ring communication paths are scheduled in parallel or interleaved during the reduction distribution phase and the full collection phase. At least two physical communication links can carry the communication load simultaneously or interleaved. Compared with the serial execution of full reduction communication on each ring communication path, the overall completion time of communication can be shortened, and the overall utilization rate of multiple physical communication links can be improved within the communication time window. The idle waiting interval of physical communication links is reduced, and the overall completion time of full reduction communication is further compressed.

[0032] According to a second aspect of the present disclosure, a collection communication device is provided, the device comprising: The ring communication path determination module is used to traverse multiple transmission speeds in descending order, determine the currently traversed transmission speed as the target transmission speed, and determine at least two ring communication paths formed by the physical communication links between multiple computing devices based on the target transmission speed. Here, the multiple transmission speeds represent the expected communication rate of the ring communication path to be constructed, each ring communication path includes multiple computing devices, and no two ring communication paths share the same physical communication link. The data fragment determination module is used to determine the data fragments in the data to be reduced that correspond to at least two ring communication paths respectively; The full reduction communication module is used to perform full reduction communication on data fragments on at least two ring communication paths to obtain the full reduction result corresponding to the data to be reduced.

[0033] According to a third aspect of the present disclosure, an electronic device is provided, including: a processor and a memory, wherein computer-readable instructions are stored in the memory, and when executed by the processor, the computer-readable instructions implement the collection communication method as described in the first aspect.

[0034] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the collection communication method as described in the first aspect.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0037] Figure 1 The schematic diagram illustrates a flow chart of a collection communication method according to some embodiments of the present disclosure.

[0038] Figure 2 The schematic diagram illustrates a fully interconnected configuration of eight GPUs according to some embodiments of the present disclosure.

[0039] Figure 3 A schematic diagram of a ring communication path 0 according to some embodiments of the present disclosure is shown.

[0040] Figure 4 A schematic diagram of a ring communication path 1 according to some embodiments of the present disclosure is shown.

[0041] Figure 5 A schematic diagram of a ring communication path 2 according to some embodiments of the present disclosure is shown.

[0042] Figure 6 A schematic diagram of a ring communication path 3 according to some embodiments of the present disclosure is shown.

[0043] Figure 7 A schematic diagram of a ring communication path 4 according to some embodiments of the present disclosure is shown.

[0044] Figure 8 A schematic diagram of a ring communication path 5 according to some embodiments of the present disclosure is shown.

[0045] Figure 9 A schematic diagram of a ring communication path 6 according to some embodiments of the present disclosure is shown.

[0046] Figure 10 The illustration schematically shows a process diagram of constructing at least two ring communication paths based on a target transmission speed and physical communication links between multiple computing devices, according to some embodiments of the present disclosure.

[0047] Figure 11 A schematic diagram of a collection communication device according to some embodiments of the present disclosure is shown.

[0048] Figure 12 The schematic diagram illustrates the structural schematic of a computer system of an electronic device according to some embodiments of the present disclosure.

[0049] Figure 13 A schematic diagram of a computer-readable storage medium according to some embodiments of the present disclosure is shown.

[0050] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0051] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0052] Furthermore, the accompanying drawings are for illustrative purposes only and are not necessarily drawn to scale. The block diagrams shown in the drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0053] In this example embodiment, a collective communication method is first provided. This method can be executed by an electronic device with collective communication scheduling and data processing capabilities, specifically applicable to servers, workstations, computing nodes, computing devices participating in collective communication, or other processing devices capable of running collective communication programs. The electronic device can be configured with a processor, memory, and a collective communication program to traverse multiple transmission speeds in descending order, determine the currently traversed transmission speed as the target transmission speed, and, based on the target transmission speed, determine at least two ring communication paths formed by physical communication links between multiple computing devices. The multiple transmission speeds represent the expected communication rate of the ring communication paths to be constructed, each ring communication path includes multiple computing devices, and no two ring communication paths share the same physical communication link. Data fragments in the data to be reduced are determined, each corresponding to one of the at least two ring communication paths. Full reduction communication is performed on the data fragments on the at least two ring communication paths to obtain the full reduction result corresponding to the data to be reduced. In this way, the data to be regulated is carried by at least two ring communication paths. Compared with the method of carrying all the data to be regulated by a single ring communication path, the amount of data that a single ring communication path needs to transmit in multiple rounds of communication can be reduced, thereby shortening the overall completion time of the full-revision communication.

[0054] Figure 1 A schematic flowchart illustrating a collection communication method according to some embodiments of the present disclosure is shown. Reference Figure 1 As shown, the collection communication method may include the following steps: Step S110: Traverse multiple transmission speeds in descending order, determine the currently traversed transmission speed as the target transmission speed, and based on the target transmission speed, determine at least two ring communication paths formed by the physical communication links between multiple computing devices; wherein, the multiple transmission speeds respectively represent the expected communication rate of the ring communication path to be constructed, each ring communication path includes multiple computing devices, and no two ring communication paths share the same physical communication link. Step S120: Determine the data fragments in the data to be standardized that correspond to at least two ring communication paths respectively; Step S130: Perform full reduction communication on the data fragments on at least two ring communication paths to obtain the full reduction result corresponding to the data to be reduced.

[0055] According to the collective communication method in this example embodiment, multiple transmission speeds are traversed in descending order to determine the currently traversed transmission speed as the target transmission speed. Based on the target transmission speed, at least two ring communication paths formed by physical communication links between multiple computing devices are determined. The multiple transmission speeds represent the expected communication rate of the ring communication paths to be constructed. Each ring communication path includes multiple computing devices, and no two ring communication paths share the same physical communication link. Data fragments corresponding to at least two ring communication paths are segmented in the data to be reduced, and full reduction communication is performed on the corresponding ring communication paths, so that the data to be reduced is carried by at least two ring communication paths respectively. Therefore, compared with the existing Ring Allreduce method where all the data to be reduced is carried by a single logical ring communication path, the amount of data that needs to be transmitted by a single ring communication path during the full reduction communication process can be reduced, thereby reducing the total data transmission time of the full reduction communication process and reducing communication latency.

[0056] In the traditional Ring Allreduce method, each computing device uses only one sending link and one receiving link in each round of communication. Even if the system hardware supports parallel direct links between multiple computing devices (such as a fully interconnected architecture), these additional links cannot be utilized simultaneously, resulting in the inability to leverage the multi-link parallel capability. Furthermore, in systems with multi-link parallel capability, most physical links remain idle for extended periods, leading to low utilization of physical link resources and limiting overall communication efficiency.

[0057] In this embodiment, under the Full Mesh architecture, multiple non-overlapping ring communication paths can be constructed to enable multiple physical communication links to be active simultaneously, effectively avoiding the link idle problem in the single Ring scheme, thereby significantly improving the overall link utilization.

[0058] Furthermore, the existing Ring Allreduce constructs only one logical loop, with all data transmitted serially along the same loop direction. This limits communication parallelism, preventing the hardware topology advantage from being translated into improved communication performance. Moreover, because all data blocks in a single Ring structure are transmitted along the same logical path, and only one valid data flow direction exists at any given time, local link congestion easily occurs under high load conditions. This concentrates the bandwidth bottleneck on a single communication path, lacking multi-path parallel transmission and load-sharing mechanisms, thus limiting the scalability of communication performance.

[0059] In this embodiment, under a Full Mesh architecture, each computing device can communicate directly with multiple other computing devices. Multiple non-shared physical communication links can simultaneously carry communication traffic, and multiple ring communication paths can execute in parallel. The data load of each ring communication path is distributed, and multiple ring communication paths can transmit data in parallel. Therefore, it can effectively improve the aggregate bandwidth in the full protocol communication process, making the communication performance closer to the theoretical peak of the hardware interconnect architecture. Furthermore, even under high load conditions, multiple parallel ring communication paths can evenly distribute the data load, improve link utilization, reduce local link congestion, and achieve scalability of communication performance. Simultaneously, it supports each ring communication path to independently execute Reduce-Scatter and Allgather operations, ensuring data correctness and high throughput during parallel communication of multiple ring communication paths, fundamentally solving the communication bottleneck problem caused by single-ring serial transmission.

[0060] Furthermore, as computing device (GPU) interconnects evolve from traditional PCIe architectures to multi-channel direct-connect architectures, hardware offers higher parallel communication capabilities. However, the existing Ring Allreduce still uses a single-path serial transmission model, resulting in the algorithm's performance not being able to fully utilize the new high-bandwidth, low-latency interconnect architecture. This leads to a mismatch between algorithm capabilities and hardware capabilities, making it difficult to adapt to the new high-bandwidth, low-latency interconnect architecture.

[0061] In this embodiment, multiple direct links under the Full Mesh architecture can be transformed into communication paths that can be used in parallel, achieving deep matching between the algorithm structure and the hardware topology, avoiding waste of hardware capabilities, and thus giving full play to the advantages of hardware interconnection.

[0062] Furthermore, based on the basic communication model of Ring Allreduce, it does not rely on a centralized scheduling node and is easy to integrate into existing collection communication frameworks. While ensuring controllable implementation complexity, it can significantly improve communication performance and has good scalability and engineering feasibility.

[0063] In addition, the above-mentioned ensemble communication method can be applied to fully interconnected GPU systems within servers, high-bandwidth and low-latency interconnected environments, and high-frequency Allreduce operations in large-scale model training, providing an efficient and scalable communication solution for practical high-performance computing.

[0064] The collection communication method in this example embodiment will be further described below.

[0065] In step S110, multiple transmission speeds are traversed in descending order, and the currently traversed transmission speed is determined as the target transmission speed. Based on the target transmission speed, at least two ring communication paths formed by the physical communication links between multiple computing devices are determined. The multiple transmission speeds represent the expected communication rate of the ring communication path to be constructed. Each ring communication path includes multiple computing devices, and no two ring communication paths share the same physical communication link.

[0066] The computing devices refer to the processing devices participating in this collective communication, which can be graphics processing units (GPUs), accelerators, or other processing units with data computing and data communication capabilities. For example, multiple computing devices could be 7 GPUs, 8 accelerators, or 10 GPUs, etc.

[0067] A physical communication link refers to a communication link formed by hardware interconnections used to transmit data between different computing devices. This hardware interconnection refers to the connection relationship between multiple computing devices at the hardware level.

[0068] In some embodiments, the physical communication link can be a direct link between two computing devices. In other embodiments, the physical communication link can also be a link in the hardware topology that characterizes the actual communication relationship between the two computing devices.

[0069] For example, if two computing devices have a direct link, then that direct link can serve as the physical communication link between the two computing devices. Similarly, if two computing devices are not directly connected but are both connected to the same PCIe switch chip and can transmit data via that PCIe switch chip, then the link in the hardware topology that represents the actual communication relationship between the two computing devices can also serve as the physical communication link between them.

[0070] A ring communication path refers to a closed communication path formed by multiple computing devices connected sequentially in a certain order and linked by physical communication links between them. In a ring communication path, the last computing device in the connection sequence can connect back to the first computing device via a physical communication link.

[0071] For example, multiple computing devices include GPU0, GPU1, GPU2, and GPU3. If GPU0, GPU1, GPU2, and GPU3 communicate sequentially in the order GPU0→GPU1→GPU2→GPU3→GPU0, then this communication path forms a circular communication path. In this circular communication path, the last computing device, GPU3, can connect back to the first computing device, GPU0; therefore, this communication path is a closed communication path.

[0072] It is understandable that physical communication links are communication resources at the hardware topology level, used to indicate which hardware link different computing devices can use to transmit data, while a ring communication path is a logical communication path determined based on one or more physical communication links.

[0073] It is understandable that for any given ring communication path, all computing devices participating in full-protocol communication are located within that ring communication path and can participate in the transmission and reduction of corresponding data fragments according to the communication order determined by that ring communication path. For example, multiple computing devices include GPU0, GPU1, GPU2, and GPU3. One ring communication path could be GPU0→GPU1→GPU2→GPU3→GPU0, and another ring communication path could be GPU0→GPU3→GPU2→GPU1→GPU0. Both of these ring communication paths include GPU0, GPU1, GPU2, and GPU3, and therefore can cover all computing devices participating in full-protocol communication.

[0074] It is understandable that no two ring communication paths share the same physical communication link; that is, different ring communication paths do not share the same physical communication link. No two ring communication paths share the same physical communication link. Therefore, the communication bandwidth of different physical communication links can be fully utilized during communication, thereby improving the performance of aggregated communication and reducing communication latency. For example, if a ring communication path uses the physical communication link from GPU0 to GPU1, then other ring communication paths will not use this physical communication link from GPU0 to GPU1.

[0075] In this embodiment of the disclosure, the physical communication links between computing devices are distinguished according to the data transmission direction and are jointly determined by the sending computing device and the receiving computing device. For example, the physical communication link used to transmit data from GPU0 to GPU1 and the physical communication link used to transmit data from GPU1 to GPU0 are two physical communication links that can be used independently.

[0076] No two ring communication paths share the same physical communication link. This means that no two ring communication paths have a physical communication link where both the sending and receiving computing devices are identical. For example, if one ring communication path uses a physical communication link from GPU0 to GPU1, then other ring communication paths will not use the physical communication link from GPU0 to GPU1, but they can use the physical communication link from GPU1 to GPU0.

[0077] In other embodiments, each computing device has the capability to use multiple physical communication links simultaneously, and each computing device uses different physical communication links in different ring communication paths. It is understood that each computing device can simultaneously transmit data with other computing devices through multiple physical communication links within the same time period. Specifically, each computing device can interconnect with other computing devices through multiple independent physical communication links, and these physical communication links can simultaneously carry data transmission on different ring communication paths; computing devices can send or receive data to different computing devices in the same clock cycle or time slice, thereby supporting parallel execution of full-protocol communication on multiple ring communication paths, improving overall communication throughput and parallelism, and reducing the impact of single-channel startup latency on performance. For example, if GPU-A is connected to GPU-B and GPU-C by two independent links each, GPU-A can simultaneously transmit data to GPU-B on one physical communication link and to GPU-C on another physical communication link.

[0078] In some embodiments, at least two pre-built ring communication paths (MULTIRING) can be obtained.

[0079] In other embodiments, at least two ring communication paths can be built based on the target server's Full Mesh architecture, without relying on cross-node switches or centralized scheduling.

[0080] In some implementations, at least two ring communication paths can be constructed using a predefined algorithm or a search algorithm. The predefined algorithm can be defined as needed. The search algorithm can be, for example, an automatic search algorithm for topology-aware ring communication paths, and is not limited to this.

[0081] In other implementations, physical communication link information between multiple computing devices can be obtained, the connection order between multiple computing devices can be determined based on the physical communication link information between multiple computing devices, and at least two ring communication paths can be constructed based on the device connection order between multiple computing devices.

[0082] Physical communication link information refers to information describing the hardware connection relationships between multiple computing devices. This information may include the two computing devices connected by the physical communication link, whether the physical communication link can be used for this aggregated communication, and link identifiers used to distinguish different physical communication links. Link identifiers are identification information used to differentiate between different hardware communication links.

[0083] Device connection order refers to the arrangement of multiple computing devices in a ring communication path. It can be understood that for a given device connection order, if it includes multiple computing devices participating in this full protocol communication, there are corresponding physical communication links between adjacent computing devices, and there is also a corresponding physical communication link between the last computing device and the first computing device in the connection order, then a ring communication path can be determined based on this device connection order.

[0084] In some implementations, after determining the multiple computing devices participating in the full protocol communication and the physical communication links between them, a pre-configured path is read; the path node devices in the ring communication path to be constructed are sequentially read from the path table, and a ring communication path is generated based on the physical communication links between each path node device. The path table records the arrangement order of the multiple computing devices in each ring communication path. Each ring communication path passes through multiple computing devices sequentially according to the corresponding arrangement order and returns from the last computing device to the first computing device to form a closed communication path.

[0085] In other implementations, at least two ring communication paths can be constructed using a topology search approach. Specifically, a topology graph can be constructed based on multiple computing devices and the physical communication links between them, where nodes in the topology graph represent computing devices and edges represent physical communication links; at least two ring communication paths are then searched from this topology graph.

[0086] In other embodiments, other methods can be used to determine at least two ring communication paths formed by the physical communication links between multiple computing devices, which are not limited here. It should be noted that in some embodiments of this disclosure, multiple transmission speeds can be traversed in descending order, the currently traversed transmission speed can be determined as the target transmission speed, and at least two ring communication paths can be constructed based on the target transmission speed and the physical communication links between multiple computing devices. For the specific construction method, please refer to the description of steps S1010 to S1030 below.

[0087] In some embodiments, such as Figure 2 As shown, the same target server includes 8 GPUs (computing devices), namely GPU0, GPU1, GPU2, GPU3, GPU4, GPU5, GPU6 and GPU7. Any two of the 8 GPUs have a physical communication link that allows them to communicate directly, forming a Full Mesh interconnect architecture.

[0088] For these 8 GPUs, 7 ring communication paths can be built on the same physical interconnect network: ring communication path 0, ring communication path 1, ring communication path 2, ring communication path 3, ring communication path 4, ring communication path 5, and ring communication path 6. Each ring communication path includes 8 GPUs, and no two ring communication paths share the same physical communication link. The 7 ring communication paths together constitute the MULTI RING communication structure.

[0089] like Figure 3 The diagram shows ring communication path 0: GPU0→GPU1→GPU2→GPU3→GPU4→GPU5→GPU6→GPU7→GPU0. Specifically, in... Figure 3 In this context, the ring communication path 0 is formed in the order of GPU0, GPU1, GPU2, GPU3, GPU4, GPU5, GPU6, GPU7, and then GPU0. When selecting GPU1 as the next path node device after GPU0, the physical communication link corresponding to GPU1 is the physical communication link used to connect GPU0 and GPU1 and form the ring communication path 0. Similarly, when selecting GPU2 as the next path node device after GPU1, the physical communication link corresponding to GPU2 is the physical communication link used to connect GPU1 and GPU2 and form the ring communication path 0. This process continues. After selecting GPU7, a physical communication link is also determined to connect GPU7 to the initial path node device GPU0 to form a closed ring communication path 0. Other physical communication links connected to each GPU but not used to form the ring communication path 0 are not considered physical communication links corresponding to the path node devices described here.

[0090] like Figure 4 The following is the ring communication path 1: GPU0→GPU2→GPU4→GPU6→GPU1→GPU3→GPU7→GPU5→GPU0.

[0091] like Figure 5 The following is the ring communication path 2: GPU0→GPU3→GPU5→GPU7→GPU1→GPU4→GPU2→GPU6→GPU0.

[0092] like Figure 6 The following is the ring communication path 3: GPU0→GPU4→GPU7→GPU2→GPU1→GPU6→GPU5→GPU3→GPU0.

[0093] like Figure 7 The following is the ring communication path 4: GPU0→GPU5→GPU1→GPU7→GPU6→GPU4→GPU3→GPU2→GPU0.

[0094] like Figure 8 The following is the ring communication path 5: GPU0→GPU6→GPU3→GPU1→GPU5→GPU2→GPU7→GPU4→GPU0.

[0095] like Figure 9 The following is the ring communication path 6: GPU0→GPU7→GPU3→GPU6→GPU2→GPU5→GPU4→GPU1→GPU0.

[0096] In each ring communication path, data can be sequentially transferred between GPUs according to the path's order, with reduction and collection operations completed within each path. Each ring communication path can independently execute full-reduction communication. By constructing multiple ring communication paths and ensuring that no two paths share the same physical communication link, physical communication link utilization can be improved, communication parallelism increased, and the impact of single-channel startup latency on overall communication performance reduced.

[0097] In step S120, data fragments in the data to be standardized that correspond to at least two ring communication paths are determined.

[0098] The data to be reduced refers to the data participating in the full reduction communication. This data can be gradient data, parameter data, intermediate calculation results, or other data that needs to be reduced between multiple computing devices during model training.

[0099] Data fragmentation refers to the data portion divided from the data to be reduced. Each data fragment corresponds to a portion of the data to be reduced. The data range of this fragment can be determined by data offset, data length, data block number, or other information that can identify the data location. Multiple data fragments collectively correspond to the data content of the data to be reduced that needs to participate in the full reduction communication. The granularity of data fragmentation can include one of the following: data chunk, data block, and data segment.

[0100] In some embodiments, the number of ring communication paths is the same as the number of data fragments.

[0101] In other embodiments, the number of ring communication paths and the number of data fragments can also be different; for example, the number of data fragments can be greater than the number of ring communication paths. When the number of data fragments can be greater than the number of ring communication paths, a corresponding number of data fragments can be allocated according to the bandwidth of each ring communication path. For example, if the bandwidth of a ring communication path is large, two or three data fragments can be allocated; if the bandwidth of a ring communication path is small, one data fragment can be allocated.

[0102] In some implementations, the data to be standardized can be divided into at least two data fragments according to the number of ring communication paths, with one ring communication path corresponding to one data fragment.

[0103] In other implementations, the data to be reduced can first be divided into multiple data fragments based on its range; then, the correspondence between each data fragment and each ring communication path can be determined based on at least two ring communication paths. For example, the data to be reduced can be divided into a first data fragment and a second data fragment, with the first data fragment corresponding to a first ring communication path and the second data fragment corresponding to a second ring communication path. If more ring communication paths exist, the correspondence between more data fragments and more ring communication paths can be determined.

[0104] In other embodiments, a correspondence between each ring communication path and a data fragment can be established. Based on this correspondence, full reduction communication can be performed on the corresponding data fragments on at least two ring communication paths to obtain the full reduction result corresponding to the data to be reduced. Here, the correspondence refers to the information used to represent the association between the data fragment and the ring communication path, which can be represented by fragment number, path number, mapping table or other data structure.

[0105] In step S130, full reduction communication is performed on the data fragments on at least two ring communication paths to obtain the full reduction result corresponding to the data to be reduced.

[0106] Full reduction communication refers to the communication process that simultaneously performs reduction operations and data distribution operations among multiple computing devices. Specifically, each computing device holds a data fragment to be reduced. Through a ring communication path or a multi-ring parallel approach, each device first performs reduction operations on its own data fragment, such as summation, finding the maximum value, or calculating the average value. Then, the reduction results are distributed to all participating computing devices in a predetermined order, allowing each device to ultimately obtain the integrated global computation result. Full reduction communication can be executed in parallel on multiple rings to fully utilize the physical communication links in a Full Mesh architecture, improving communication parallelism and bandwidth utilization.

[0107] The allreduce result refers to the final data result obtained after allreduce communication processing. In this allreduce result, each computing device contains a global reduced value for the original data to be reduced; that is, it integrates the unified result of the reduction operation on the corresponding data fragments across multiple computing devices. The allreduce result ensures data correctness and facilitates direct use by subsequent computing tasks without requiring additional data integration operations.

[0108] It is understandable that, since each ring communication path includes multiple computing devices participating in this full-reduction communication, each ring communication path can complete the data transmission and reduction processing of the corresponding data fragments among the multiple computing devices it covers, and obtain the full-reduction result.

[0109] For any ring communication path, corresponding data fragments can be transmitted between multiple computing devices along the ring communication path, and during the transmission process, reduction operations are performed on the corresponding data from different computing devices to enable the corresponding data fragments to complete full-reduction communication.

[0110] During full protocol communication, the full protocol communication of the same data fragment is completed on its corresponding ring communication path, and different data fragments are carried by different ring communication paths. Since no two ring communication paths share the same physical communication link, the physical communication links corresponding to different data fragments are distinguished from each other when performing full protocol communication. Therefore, the transmission of data to be regulated and the regulated task can be distributed across at least two ring communication paths.

[0111] In some implementations, full reduction communication can be performed in a pipelined manner on at least two ring communication paths to obtain the full reduction result corresponding to the data to be reduced.

[0112] In some implementations, full reduction communication can be performed on data fragments in parallel on at least two ring communication paths to obtain the full reduction result corresponding to the data to be reduced.

[0113] Specifically, each computing device performs sending and receiving operations simultaneously through multiple physical communication links to support parallel communication via multiple ring communication paths. Each computing device supports efficient data transmission via GPU Direct RDMA technology and can utilize multiple Streams or QueuePairs (QPs) to process multiple communication tasks in parallel simultaneously. Furthermore, during full protocol communication, the underlying communication interface type is not limited, allowing adaptation to different hardware platforms and interconnect protocols, thus ensuring the universality and scalability of multi-link parallel communication.

[0114] In some embodiments, for each ring communication path, the communication and local reduction computation of the ring communication path can be controlled to be processed in parallel. Each computing device can simultaneously perform send and receive operations on multiple ring communication paths, as well as perform parallel reduction processing on multiple data fragments.

[0115] In some implementations, on each ring communication path, a reduce-scatter phase is sequentially performed on the corresponding data fragments. Each computing device performs a reduce operation on the received data block and its corresponding local data block. After completing this round of data computation, the reduce result is sent to the next computing device in the ring communication path. After each ring communication path completes the reduce-scatter operation, an allgather phase is sequentially performed. Each computing device receives the reduce result data blocks from other computing devices in the ring order and integrates its local data blocks to obtain the complete full reduce result. The full reduce communication on each ring communication path is independent in time and can be started sequentially or scheduled as needed, but within each ring, the order of reduce-scatter followed by allgather is maintained.

[0116] In other implementations, reduction scattering and full collection operations are performed on corresponding data fragments along multiple ring communication paths, while parallel execution across multiple rings is enabled. Each computing device can simultaneously send and receive data blocks via multiple physical communication links, with data fragments from multiple rings interleaved across different links. The reduction scattering and full collection phases can be interleaved between different rings, and data reception and transmission are pipelined, thereby achieving parallelization of full-reduction communication and maximizing bandwidth utilization. Each computing device merges the reduction results from different ring communication paths (by splicing or reassembling them according to the data fragment order) to obtain a complete full-reduction result.

[0117] In other implementations, other methods may be used to perform full protocol communication, which are not limited here.

[0118] The contents of steps S110 to S130 will be described in detail below.

[0119] In one example embodiment of this disclosure, determining at least two ring communication paths formed by physical communication links between multiple computing devices based on a target transmission speed includes: determining multiple computing devices participating in full protocol communication, and physical communication links between the multiple computing devices; and constructing at least two ring communication paths based on the target transmission speed and the physical communication links between the multiple computing devices.

[0120] In some implementations, a topology map of the target server can be obtained, from which multiple computing devices participating in full protocol communication and the physical communication links between these devices can be identified. Since all the computing devices reside on the same target server (the same node), physical communication links between them are guaranteed. Furthermore, a physical communication link exists between any two computing devices participating in full protocol communication within the target server. This topology map is also the fully interconnected map of the Full Mesh architecture.

[0121] When constructing a ring communication path, a computing device is first selected from multiple computing devices as the starting node device. Then, the next hop computing device is selected sequentially based on the availability of the physical communication link, so that each path passes through these multiple computing devices in sequence. Finally, the physical communication link between the last hop computing device and the starting node device is connected to construct a complete ring communication path.

[0122] To fully utilize hardware resources, it can be ensured that the physical communication links used by different ring communication paths between any adjacent node devices are not duplicated; alternatively, while allowing different ring communication paths to use the same physical communication link in a time-sharing manner, it can be ensured that different ring communication paths do not use the same physical communication link simultaneously within the same time window, thereby avoiding link usage conflicts between different ring communication paths and forming at least two independent ring communication paths. Each ring communication path covers all computing devices participating in the full-reduction communication and can be used to execute its own data sharding full-reduction communication to ensure communication load balancing, maximize bandwidth utilization, and improve overall Allreduce performance.

[0123] The same time window refers to the same communication period used to schedule data transmission for at least two ring communication paths. In some embodiments, the same time window can be a pre-divided time slice, an execution period corresponding to a communication round, or a period in which the communication execution times of at least two ring communication paths overlap. Within the same time window, if a physical communication link has been assigned to one of the ring communication paths, then that physical communication link will not be assigned to other ring communication paths within that time window; after the end of that time window, the physical communication link can be assigned to other ring communication paths in a subsequent time window that does not overlap with that time window.

[0124] For example, within the first time window, if the first ring communication path uses the physical communication link from GPU0 to GPU1 to perform data transmission, then other ring communication paths will not use the physical communication link from GPU0 to GPU1 within the first time window. In the second time window after the first time window ends, if the first ring communication path no longer uses the physical communication link, then the physical communication link from GPU0 to GPU1 can be allocated to the second ring communication path. Therefore, although different ring communication paths can reuse the same physical communication link in different time windows, they will not simultaneously occupy the same physical communication link within the same time window.

[0125] After obtaining a ring communication path, another ring communication path can be built based on the physical communication links between multiple computing devices, until at least two ring communication paths are obtained.

[0126] In other implementations, multiple transmission speeds are traversed in descending order, and the currently traversed transmission speed is determined as the target transmission speed. Based on the target transmission speed and the physical communication links between the multiple computing devices, at least two ring communication paths are constructed. Specifically, multiple transmission speeds corresponding to the ring communication path to be constructed can be obtained, where each transmission speed represents the expected communication rate of the ring communication path to be constructed; multiple transmission speeds are traversed in descending order, and the currently traversed transmission speed is determined as the target transmission speed; based on the target transmission speed and the physical communication links between the multiple computing devices, at least two ring communication paths are constructed.

[0127] In some embodiments, the multiple different transmission speeds can be multiple different discrete speed values ​​that are preset. For example, three discrete transmission speeds can be set: 100 GB / s, 80 GB / s, and 60 GB / s.

[0128] In other embodiments, multiple different transmission speeds can be determined based on the capabilities of the physical communication links between multiple computing devices. Specifically, the bandwidth of each physical communication link can be obtained, and multiple different transmission speeds can be determined based on the bandwidth of each physical communication link. The bandwidth of each physical communication link can be, for example, the theoretical maximum bandwidth of each physical communication link or the currently available bandwidth, and is not limited to these.

[0129] For example, in a fully interconnected system with 8 GPUs, if the link bandwidths are 50 GB / s, 45 GB / s, and 40 GB / s, these bandwidth values ​​can be directly used as candidate transmission speeds, or a set of discrete speeds can be generated based on these values, such as 50 GB / s, 45 GB / s, 40 GB / s, and 35 GB / s, to adapt to the construction requirements of different ring paths.

[0130] The target transmission speed is the transmission speed currently being traversed and used in the current ring communication path construction process. Traversing multiple transmission speeds in descending order means first selecting the highest transmission speed among the multiple speeds as the target transmission speed used in the current construction process, and then continuing to select other transmission speeds in descending order. For example, if multiple transmission speeds include 50GB / s, 45GB / s, and 40GB / s, then the traversed 50GB / s will be used as the target transmission speed to construct at least two ring communication paths. If 50GB / s cannot construct at least two ring communication paths, then the next transmission speed in descending order, 45GB / s, will be traversed as the target transmission speed, and at least two ring communication paths will be constructed using this target speed of 45GB / s, and so on.

[0131] In some implementations, for each target transmission speed, a ring channel can be constructed using a depth-first search (DFS) + backtracking algorithm. Specifically, after determining the target transmission speed, this target speed can be used as a constraint parameter for constructing the ring communication path, and at least two ring communication paths can be constructed in conjunction with the physical communication links between multiple computing devices. Therefore, when constructing at least two ring communication paths, not only is the existence of physical communication links between multiple computing devices considered, but also the ring communication path for full-representation communication is determined based on the current target transmission speed. This allows for the priority use of higher transmission speeds among multiple transmission speeds to construct at least two ring communication paths, ensuring that the construction process matches the communication capabilities supported by the physical communication links between multiple computing devices. If the construction conditions corresponding to the current target transmission speed cannot be met, path construction can continue based on lower transmission speeds.

[0132] Understandably, in server topologies containing multiple computing devices, high-bandwidth physical communication links are highly clustered. Therefore, a hierarchical decreasing speed strategy can be adopted. This involves traversing multiple transmission speeds in descending order and using the currently traversed speed as the target speed to construct at least two ring communication paths. This prioritizes the consumption of high-bandwidth resources and yields a globally optimal combination of at least two ring communication paths. Furthermore, experiments show that this descending order of transmission speeds for constructing ring communication paths allows the search for ring communication paths to converge to near the optimal solution within a very small number of backtracking steps, resulting in a globally optimal combination of at least two ring communication paths.

[0133] In some implementations, such as Figure 10 As shown, based on the target transmission speed and the physical communication links between multiple computing devices, at least two ring communication paths are constructed, including the following steps S1010 to S1030: Step S1010: Select path node devices in the ring communication path to be constructed from multiple computing devices hop by hop, and deduct the bandwidth usage for this step from the remaining bandwidth of the physical communication link corresponding to the path node device.

[0134] In this context, a path node device refers to a computing device selected to join the ring communication path to be constructed. Hop-by-hop selection refers to sequentially selecting computing devices to form the ring communication path according to its connection direction. Each selected path node device allows for the determination of the physical communication link between that path node device and its adjacent, already joined path node devices.

[0135] Remaining bandwidth is the bandwidth obtained by subtracting the allocated bandwidth from the total bandwidth of the physical communication link. Total bandwidth refers to the data transmission capacity that the physical communication link can provide at the hardware level. Allocated bandwidth refers to the bandwidth already allocated from the total bandwidth of the physical communication link during the determination of the ring communication path. Bandwidth occupancy refers to the link bandwidth required for a ring communication path to communicate at the target transmission speed.

[0136] For example, assuming the total bandwidth of the physical communication link from computing device GPU-A to computing device GPU-B is 100GB / s, the ring communication path to be built currently requires 50GB / s of link bandwidth when communicating at the target transmission speed. If this physical communication link is not used by other ring communication paths executing within the current time window, and its remaining bandwidth is not less than 50GB / s, 50GB / s can be deducted from the remaining bandwidth of this physical communication link during the construction of the current ring communication path.

[0137] If the current ring communication path fails to be constructed and a rollback is executed, the bandwidth usage deducted in this instance will be restored. If the current ring communication path is successfully constructed, in an implementation where no two ring communication paths share the same physical communication link, that physical communication link will no longer be used to construct other ring communication paths; in an implementation where time multiplexing is used for different ring communication paths, that physical communication link will no longer be used for other ring communication paths within the current time window, but may be used for other ring communication paths in subsequent time windows that do not overlap with the current time window.

[0138] In some implementations, when selecting path node devices hop-by-hop, a computing device can be selected from multiple computing devices as the starting path node device, and then the next path node device can be selected from the computing devices that have not yet joined the ring communication path to be built. When there is a physical communication link between the selected next path node device and the already selected path node device, a communication connection can be formed in the ring communication path to be built.

[0139] In other embodiments, selecting path node devices in the ring communication path to be constructed from multiple computing devices hop-by-hop includes: determining candidate node devices that have not been added to the ring communication path to be constructed from multiple computing devices, and obtaining topology evaluation parameters of the candidate node devices; determining the evaluation results of the candidate node devices based on the topology evaluation parameters; and selecting path node devices in the ring communication path to be constructed hop-by-hop according to the evaluation results of each candidate node device.

[0140] In this context, a candidate node device refers to a computing device that is not yet included in the ring communication path to be built and can be selected as a path node device. Topology evaluation parameters are parameters used to evaluate the quality of the topology connection when a candidate node device is added to the ring communication path to be built. This topology connection quality characterizes the communication connection between the candidate node device and the current node device.

[0141] Topology evaluation parameters can include at least two of the following: inter-node bandwidth, inter-node hop count, Peripheral Component Interconnect (PCI) uplink bandwidth, intra-node bandwidth, intra-node hop count, and topology distance index. Inter-node bandwidth refers to the bandwidth available for communication between a candidate node and the current node. Inter-node hop count refers to the number of links or forwarding layers required for communication between the candidate node and the current node. PCI uplink bandwidth refers to the bandwidth available for upward communication via PCI-related links. Intra-node bandwidth refers to the communication bandwidth between different computing devices within the same server. Intra-node hop count refers to the number of links required for communication between different computing devices within the same server. Topology distance index is an indicator used to characterize the topological proximity between a candidate node and the current node.

[0142] In other embodiments, topology evaluation parameters may also include, but are not limited to, the historical utilization rate of physical communication links, the delay or jitter of physical communication links, the computing resource occupancy information of candidate node devices, energy consumption or power constraints, link reliability or error rate, etc.

[0143] Evaluation results refer to the results used to indicate the priority of candidate node devices, determined based on topology evaluation parameters. Evaluation results can be scores, ranking results, or priority indicators, and are not limited to these.

[0144] In some implementations, when determining the evaluation result based on topology evaluation parameters, each candidate node device can be evaluated according to the impact of each topology evaluation parameter on the quality of ring communication path construction. Specifically, the weighting factors of each topology evaluation parameter can be obtained, and the topology evaluation parameters can be weighted based on the weighting factors to obtain the evaluation result.

[0145] For example, the bandwidth between nodes can be configured with a higher weighting factor, while other topology evaluation parameters can be configured with lower weighting factors. Then, the topology evaluation parameters are weighted to obtain the evaluation result.

[0146] In some embodiments, after determining the evaluation results of each candidate node device, a path node device can be selected hop-by-hop from each candidate node device according to the evaluation results; for the current hop, if the remaining bandwidth of the physical communication link corresponding to the path node device selected in the current hop does not meet the bandwidth condition corresponding to the target transmission speed, a new path node device for the current hop is selected according to the evaluation results.

[0147] For example, if the evaluation result is a score, the path node devices can be selected hop by hop in descending order of score. For the current hop, the computing device with the highest score is selected first as the path node device for the current hop. If the remaining bandwidth of the physical communication link corresponding to the path node device does not meet the bandwidth condition corresponding to the target transmission speed, the computing device with the second highest score is selected as the path node device for the current hop in descending order of score. This process is repeated hop by hop to select path node devices, and then a ring communication path is constructed based on the physical communication links between each path node device.

[0148] It is understandable that in the process of selecting each hop path node device in a ring communication path, it is necessary to select the next hop path node device from multiple candidate node devices. If all possible permutations and combinations of candidate node devices are exhaustively enumerated, the computational complexity is O(N!). For a large number of candidate node devices, a lot of time is wasted, resulting in low efficiency in constructing a ring communication path.

[0149] In this embodiment, during the selection of each hop path node device in the ring communication path, the evaluation result of the candidate node device can be determined based on the topology evaluation parameters of the candidate node device. Then, the path node device can be selected hop by hop according to the evaluation result of each candidate node device. This allows the ring communication path construction process to combine the topology connection quality between multiple computing devices and prioritize the selection of candidate node devices with higher topology connection quality, thereby reducing invalid path attempts and improving the construction efficiency of the ring communication path.

[0150] Step S1020: If the remaining bandwidth of the physical communication links corresponding to each path node device meets the bandwidth conditions corresponding to the target transmission speed, then a ring communication path is constructed based on the physical communication links corresponding to each path node device.

[0151] The bandwidth condition corresponding to the target transmission speed refers to the amount of bandwidth available in the physical communication link that can meet the bandwidth occupancy requirement for that target transmission speed. The bandwidth condition can be that the remaining bandwidth of the physical communication link is greater than or equal to the bandwidth occupancy requirement for the target transmission speed.

[0152] For example, if the remaining bandwidth of a physical communication link is greater than or equal to the bandwidth occupancy corresponding to the target transmission speed, it means that the remaining bandwidth of the physical communication link meets the bandwidth condition corresponding to the target transmission speed; if the remaining bandwidth of a physical communication link is less than the bandwidth occupancy corresponding to the target transmission speed, it means that the remaining bandwidth of the physical communication link does not meet the bandwidth condition corresponding to the target transmission speed.

[0153] If the remaining bandwidth of the physical communication links between each path node device is greater than or equal to the bandwidth occupancy corresponding to the target transmission speed, it means that the remaining bandwidth of the physical communication links corresponding to each path node device meets the bandwidth condition corresponding to the target transmission speed.

[0154] In some implementations, if the remaining bandwidth of the physical communication links corresponding to each path node device meets the bandwidth conditions corresponding to the target transmission speed, a ring communication can be constructed according to the physical communication links between the path node devices selected hop by hop.

[0155] In other embodiments, after the path node devices selected hop-by-hop include all computing devices participating in the full protocol communication, it can be determined whether a closed ring communication path can be formed based on the connection relationship between each path node device. If there are corresponding physical communication links between adjacent path node devices arranged in the hop-by-hop selection order, and the last selected path node device can be connected to the starting path node device through a physical communication link, then a ring communication path is constructed. If there are no corresponding physical communication links between adjacent path node devices, or the last selected path node device cannot be connected to the starting path node device through a physical communication link, then the path node devices cannot form a closed ring communication path.

[0156] It is understandable that a ring communication path is formed hop by hop by hop by multiple path node devices and the physical communication links between each path node device. Since the ring communication path is constructed under the constraints of the bandwidth conditions corresponding to the physical communication links and the target transmission speed, it can avoid forming a ring communication path that cannot support the target transmission speed and improve the availability of the ring communication path.

[0157] It is understood that the physical communication link corresponding to the path node device in this embodiment refers to the physical communication link used to connect two adjacent path node devices in the hop-by-hop selection order and to form the ring communication path to be constructed during the hop-by-hop construction process, rather than all physical communication links connected to the corresponding path node device. Specifically, after selecting the path node device of the current hop, the physical communication link used to connect the path node device of the previous hop and the path node device of the current hop can be determined, and the remaining bandwidth and bandwidth usage deduction can be performed for the physical communication link; after selecting the last path node device, the physical communication link used to connect the last path node device and the starting path node device can also be determined to form a closed ring communication path.

[0158] When two adjacent path node devices communicate via a direct physical communication link, the physical communication link corresponding to each path node device can be the direct physical communication link between the two adjacent path node devices. When two adjacent path node devices communicate via a target path composed of multiple physical communication links, the physical communication links corresponding to each path node device can include each physical communication link that makes up the target path, and the remaining bandwidth can be determined and the bandwidth usage can be deducted for each physical communication link separately.

[0159] In other embodiments, the method further includes: determining a target path between different computing devices from a plurality of computing devices; selecting path node devices in a ring communication path to be constructed hop-by-hop from the plurality of computing devices, including: selecting path node devices in a ring communication path to be constructed hop-by-hop from the plurality of computing devices based on the target path between different computing devices.

[0160] The target path refers to the communication path determined between two different computing devices. When constructing a ring communication path, the target path is used to determine the data transmission path between two adjacent path node devices. The target path can be formed by a single physical communication link between the two computing devices, or by multiple physical communication links connected sequentially. If the target path is formed by multiple physical communication links, then all of these physical communication links belong to the link resources corresponding to the target path.

[0161] When selecting path node devices hop by hop, based on the path node devices that have already joined the ring communication path to be built, it is possible to query whether there is a target path between the path node device and the computing device that has not joined the ring communication path to be built; if there is a target path, the computing device that has not joined can be used as a candidate node device, and then the path device can be selected hop by hop from each candidate node device.

[0162] The step of selecting path devices from each candidate node device hop by hop may include: obtaining topology evaluation parameters of the candidate node devices; determining the evaluation results of the candidate node devices based on the topology evaluation parameters; and selecting path node devices in the ring communication path to be constructed hop by hop according to the evaluation results of each candidate node device.

[0163] For example, if computing device A is currently a path node device already added to the ring communication path to be built, the target path between computing device A and other unadded computing devices can be queried. If a target path exists between computing device A and computing device B, then computing device B can be selected as a candidate for the next path node device, and the communication connection between them in the ring communication path to be built can be determined based on the target path between computing device A and computing device B. In this way, multiple path node devices in the ring communication path to be built can be selected hop-by-hop.

[0164] It is understandable that by determining the target path between different computing devices through the topology path pre-computation stage, and then selecting path node devices hop by hop based on the target path through the ring communication path search stage, at least two ring communication paths can be constructed more quickly, reducing communication latency.

[0165] In other embodiments, determining a target path between different computing devices from a plurality of computing devices includes: for a first device and a second device among the plurality of computing devices, determining at least two candidate paths between the first device and the second device, and determining the bottleneck bandwidth of each candidate path; the bottleneck bandwidth is the minimum bandwidth of each physical communication link in the candidate path; if the bottleneck bandwidths of the candidate paths are different, then determining the target path from the at least two candidate paths based on the bottleneck bandwidths; if the bottleneck bandwidths of the candidate paths are the same, then determining the hop count of each candidate path, and determining the target path from the candidate paths based on the hop counts.

[0166] Here, the first device and the second device are any two different computing devices among a plurality of computing devices. The first device and the second device are used to represent two endpoint devices for which the target path needs to be determined, and are not limited to fixed numbers or fixed positions among the plurality of computing devices.

[0167] A candidate path refers to a selectable communication path between the first device and the second device. A candidate path can include a direct physical communication link between the first and second devices, or a communication path formed by connecting multiple physical communication links. The target path is the communication path determined from at least two candidate paths, used for data transmission between adjacent path node devices during the subsequent ring communication path construction process.

[0168] Bottleneck bandwidth refers to the minimum bandwidth of all physical communication links in a candidate path. Since the overall transmission capacity of a candidate path is limited by the smaller bandwidth of the physical communication links within that path, bottleneck bandwidth can be used to characterize the effective transmission capacity of that candidate path. For a candidate path, the bandwidth of each physical communication link included in the candidate path can be determined first, and then the minimum value among them can be determined as the bottleneck bandwidth of the candidate path.

[0169] In some implementations, a breadth-first search is performed on each computing device as a first device (the source node device of the ring communication path to be constructed) to determine at least two candidate paths between the first device and the second devices (the target node devices of the ring communication path to be constructed). Specifically, a breadth-first search can be performed on the entire hardware topology graph to pre-compute candidate paths from the first device to all other second devices, thereby selecting the optimal path.

[0170] After determining the bottleneck bandwidth of each candidate path, the target path can be determined based on the bottleneck bandwidth. In some implementations, if the bottleneck bandwidths of the candidate paths are different, the candidate path with the larger bottleneck bandwidth can be determined as the target path between the first device and the second device. Here, "larger bottleneck bandwidth" can be any one of the largest, second largest, or multiple larger bottleneck bandwidths.

[0171] If the bottleneck bandwidth of all candidate paths is the same, the hop count of each candidate path can be further determined, and the candidate path with the smaller hop count can be selected as the target path. Here, "smallest hop count" can be the smallest, the second smallest, or any of several smaller hop counts. The hop count refers to the number of communication segments traversed by the candidate path from the first device to the second device, or the number of physical communication links contained in the candidate path.

[0172] In other embodiments, when a candidate path between the first device and the second device is constructed, the candidate path is recorded in a memory. If the bottleneck bandwidth of the newly constructed candidate path between the first device and the second device is found to be greater than the bottleneck bandwidth of the already recorded candidate path, the bottleneck bandwidth of the already recorded candidate path is replaced with the newly constructed candidate path. If the bottleneck bandwidth of the newly constructed candidate path and the already recorded candidate path are the same, the hop count of the newly constructed candidate path and the already recorded candidate path are determined respectively, and the candidate path with fewer hops is selected as the candidate path in the memory. After the construction is completed, the candidate path recorded in the memory is used as the target path between the first device and the second device.

[0173] Understandably, the traditional Breadth-First Search (BFS) method only pursues the shortest hop-count ring communication path. However, this embodiment introduces a bandwidth-first pruning criterion. By using bandwidth-first BFS to search the topology path, candidate paths between different computing devices are pre-calculated. If the bottleneck bandwidths of each candidate path are different, the target path is determined from at least two candidate paths based on each bottleneck bandwidth. When the bottleneck bandwidths are the same, the candidate path with the smaller hop count can be determined as the target path between the first and second devices. This ensures that the pre-calculated target paths are all the maximum bandwidth paths between the first and second devices, providing a high-quality candidate path space for subsequent ring communication path searches. It also reduces the number of communication segments traversed for data transmission between adjacent path nodes, thereby improving the communication efficiency of the communication path and reducing communication latency.

[0174] In other embodiments, pruning rules for single-hop computing device relays can be implemented during the breadth-first search propagation process. Specifically, apart from the first device (source node device) itself, the remaining computing devices can be allowed to act as one-hop direct relays; that is, each computing device cannot act as an intermediate node device in multi-hop forwarding. The pruning rules for single-hop computing device relays can be used to avoid unnecessary cross-computing device multi-hop forwarding overhead by treating computing devices as general routing nodes, while maintaining the endpoint role of computing devices in aggregated communication, consistent with the actual communication semantics, thereby improving path construction efficiency and reducing communication latency.

[0175] In other embodiments, the method further includes: when the remaining bandwidth of the physical communication link corresponding to the current hop path node device does not meet the bandwidth condition corresponding to the target transmission speed during the construction of the current ring communication path, restoring the bandwidth occupancy deducted from the physical communication link corresponding to the path node device, and reselecting the current hop path node device.

[0176] It is understandable that after selecting the path node device for the current hop from multiple computing devices, the bandwidth usage for this hop is deducted from the remaining bandwidth of the physical communication link corresponding to the path node device. The bandwidth usage for this hop can be the target bandwidth usage corresponding to the target transmission speed, or it can be a bandwidth usage lower than the target bandwidth usage.

[0177] If the remaining bandwidth of the physical communication link corresponding to the path node device is greater than or equal to the bandwidth occupancy corresponding to the target transmission speed, then the bandwidth occupancy deducted this time is the target bandwidth occupancy corresponding to the target transmission speed; if the remaining bandwidth of the physical communication link corresponding to the path node device is less than the bandwidth occupancy corresponding to the target transmission speed (the remaining bandwidth of the physical communication link is insufficient to support the current target transmission speed requirement), then the bandwidth occupancy deducted this time is the bandwidth occupancy lower than the target bandwidth occupancy, such as the remaining bandwidth of the physical communication link corresponding to the path node device, or any bandwidth occupancy less than that remaining bandwidth, etc., and is not limited to these.

[0178] Restoring the bandwidth usage deducted from the physical communication link corresponding to the path node device can be understood as canceling the bandwidth usage that has been deducted, that is, restoring the physical communication link to the remaining bandwidth state before the deduction, so that the physical communication link returns to the state of remaining bandwidth before the deduction.

[0179] In some implementations, the bandwidth usage deducted from the physical communication link corresponding to the path node device can be restored, and the system can return to the previous decision point and reselect the path node device for the current hop. The previous decision point is the decision point at which the current hop selects the path node device.

[0180] After returning to the previous hop decision point, other computing devices can be identified, and the path node device for the current hop can be selected from these other computing devices. These other computing devices can be computing devices that are not included in the ring communication path to be built and whose bandwidth requirements are not met, except for the current hop.

[0181] Specifically, other computing devices are used as candidate node devices for the current hop, and the topology evaluation parameters of the candidate node devices are obtained; based on the topology evaluation parameters, the evaluation results of the candidate node devices are determined; and according to the evaluation results of each candidate node device, the path node device for the current hop is reselected.

[0182] Understandably, the deduction-back mechanism can ensure that bandwidth resources between at least two ring communication paths are not over-allocated.

[0183] Step S1030: Based on the physical communication links that are not used by the constructed ring communication paths, continue to construct the next ring communication path until at least two ring communication paths are obtained.

[0184] After constructing a ring communication path and updating the remaining bandwidth of its occupied physical communication links, new ring communication paths can be constructed based on the physical communication links between multiple computing devices and the updated remaining bandwidth, until at least two ring communication paths are obtained. It is understood that the new ring communication path also needs to meet the bandwidth requirements corresponding to the target transmission speed in the remaining bandwidth of its included physical communication links. Furthermore, the physical communication links used by the already constructed ring communication path are no longer used to construct new ring communication paths; that is, the next ring communication path is constructed based on the physical communication links not used by the already constructed ring communication path, ensuring that no two ring communication paths share the same physical communication link.

[0185] In some embodiments, the desired number of ring communication paths to be constructed is obtained, and based on each physical communication link not used by the constructed ring communication paths, the next ring communication path is constructed until the desired number of ring communication paths are constructed.

[0186] The desired number can be set as needed. For example, if the target server includes 8 GPUs (computing devices), then the desired number of ring communication paths to be built could be 7.

[0187] In other embodiments, the method further includes: if at least two ring communication paths cannot be constructed for the currently traversed target transmission speed, then return to the step of traversing multiple transmission speeds in descending order until at least two ring communication paths are constructed.

[0188] The target transmission speed currently being traversed refers to the transmission speed that is currently selected and used to construct the ring communication path when traversing multiple transmission speeds in descending order.

[0189] In some implementations, if at least two ring communication paths cannot be constructed, multiple transmission speeds are traversed in descending order to determine the new transmission speed encountered, which is then identified as the new target transmission speed. Based on this new target transmission speed and the physical communication links between the multiple computing devices, at least two ring communication paths are constructed. The new target transmission speed refers to a transmission speed that is lower than the current target transmission speed among the multiple transmission speeds and serves as a constraint in the subsequent construction process.

[0190] For example, if only one ring communication path can be constructed based on the current target transmission speed, or if at least two ring communication paths that meet the bandwidth conditions cannot be constructed, it means that no at least two ring communication paths have been obtained under the current target transmission speed. In this case, a new target transmission speed is determined to continue constructing at least two ring communication paths.

[0191] Understandably, if at least two ring communication paths are not obtained at the current target transmission speed, a new transmission speed can be selected in descending order of multiple transmission speeds, and this new transmission speed can be determined as the new target transmission speed. Since the new target transmission speed is lower than the current target transmission speed, its corresponding bandwidth requirement is relatively reduced. Therefore, the physical communication links between multiple computing devices are more likely to meet the bandwidth requirements corresponding to the new target transmission speed.

[0192] If it is impossible to build at least two ring communication paths based on the higher target transmission speed, then continue to traverse the lower transmission speed to build at least two ring communication paths. In this way, the ring communication path construction process can take into account both the higher transmission speed and the actual bandwidth conditions of the physical communication link, thereby increasing the probability of successfully building at least two ring communication paths.

[0193] In some embodiments, when searching for a ring communication path, the solution space can be explored as much as possible under global timeout constraints by traversing different combinations of transmission speed and path type parameters (homogeneous or heterogeneous ring communication path strategies, tree-based degenerate alternative paths, etc.). Once a ring communication path that meets the conditions is found at a certain transmission speed, the optimization phase begins. Using the current solution as a lower bound, the optimization attempts to find even better path combinations at higher transmission speeds or by increasing the number of ring communication paths, thereby improving the total throughput and parallel efficiency of the fully reduced communication of multiple ring communication paths.

[0194] Furthermore, in some embodiments, the above method further includes: after constructing at least two ring communication paths, determining the total throughput of the at least two ring communication paths; and while keeping the total throughput constant, reducing the target transmission speed of the at least two ring communication paths when increasing the number of at least two ring communication paths.

[0195] Total throughput refers to the total amount of data transmitted when full protocol communication is performed through at least two ring communication paths.

[0196] In some implementations, the total throughput of at least two ring communication paths can be determined based on the target transmission speed and the number of at least two ring communication paths. Specifically, the total throughput of at least two ring communication paths can be obtained by multiplying the target transmission speed by the number of at least two ring communication paths.

[0197] In other implementations, the sub-throughputs of each ring communication path can be added together to obtain the total throughput of at least two ring communication paths.

[0198] In other embodiments, the total throughput of at least two ring communication paths can be monitored by the throughput monitoring unit, or the total throughput of at least two ring communication paths can be determined by other means, which is not limited here.

[0199] It is understandable that when increasing the number of ring communication paths, the target transmission speed of each path can be reduced accordingly in order to improve channel parallelism and optimize the communication efficiency of multiple ring paths while keeping the total throughput unchanged.

[0200] For example, if the sub-throughputs of the three ring communication paths are 40 GB / s, 40 GB / s and 20 GB / s respectively, the total throughput is 100 GB / s. When the number of ring communication paths increases to 6 (doubles), the speed of each path can be reduced proportionally so that the total throughput remains at 100 GB / s, while doubling the channel parallelism.

[0201] In some embodiments, during the construction of the ring communication path, when the target transmission speed reaches the theoretical upper limit of the hardware, the total throughput of at least two ring communication paths is determined; when the number of at least two ring communication paths is increased, the target transmission speed of at least two ring communication paths is reduced.

[0202] In some embodiments, after a complete ring communication path is successfully constructed, the search for the next ring communication path is recursively performed, and resource isolation between different ring communication paths is achieved through bandwidth deduction. The construction of the ring communication path is terminated when the construction termination condition is met. The termination condition can be set as needed, such as when the number of ring communication paths constructed reaches a preset upper limit, a global timeout occurs, or a theoretically bounded solution is found (total throughput reaches the total physical bandwidth of all physical communication links), and is not limited to these.

[0203] Understandably, the aforementioned ensemble communication method provides a topology-aware, ring-based ensemble communication path automatic search method for large-scale heterogeneous computing clusters. It can automatically discover the optimal multi-ring communication path scheme on arbitrarily complex computing device interconnection topologies, combining search efficiency with solution quality. Furthermore, it eliminates the need for manual parameter tuning, improving communication efficiency and reducing communication latency. Moreover, this optimization, without increasing physical bandwidth consumption, hides the single-channel startup latency by increasing the parallelism at the ring communication path level, significantly improving actual communication performance in small to medium message scenarios.

[0204] In other embodiments, the total throughput of at least two ring communication paths (number of ring communication paths × target transmission speed) is used as the primary indicator; a higher total throughput indicates better overall performance of the at least two ring communication paths. When the total throughput is the same, the total hop count of the at least two ring communication paths (the sum of the hop counts of the at least two ring communication paths) is used as a secondary indicator; fewer hops indicate lower end-to-end latency, thus better overall performance of the at least two ring communication paths. This dual-indicator evaluation system takes into account both bandwidth utilization and communication latency.

[0205] In some embodiments, determining data fragments in the data to be reduced that correspond to at least two ring communication paths includes: dividing the data to be reduced into multiple data fragments; determining the mapping relationship between the multiple data fragments and at least two ring communication paths; and determining the data fragments that correspond to at least two ring communication paths based on the mapping relationship.

[0206] The data to be reduced refers to the data participating in this full reduction communication. This data can be gradient data, parameter data, intermediate calculation results, or other data that needs to undergo reduction operations across multiple computing devices during model training. The data to be reduced can include multiple data elements, and different computing devices can each hold their own data to be reduced during this full reduction communication.

[0207] Data fragmentation refers to the data portion obtained from the data to be reduced. Each data fragment corresponds to a certain range of data within the data to be reduced. This data range can be determined by data offset, data length, data block number, data start position, or data end position. Multiple data fragments together cover the data content within the data to be reduced that needs to participate in the full reduction communication.

[0208] A mapping relationship refers to the information used to represent the correspondence between data fragments and ring communication paths. This mapping relationship can be represented by fragment numbers, path numbers, mapping tables, index relationships, or other data structures. Through the mapping relationship, it can be determined which ring communication path carries each data fragment, and also which data fragment each ring communication path needs to process. The mapping rules between data fragments and ring communication paths can include one of the following: static, dynamic, and round-robin.

[0209] In some embodiments, data sharding can be processed according to predetermined rules: different data shards are assigned to different ring communication paths, and Allreduce operations are executed in parallel on different ring communication paths. The size of the data shard on each ring communication path is smaller than the data size of a single Ring Allreduce operation; the communication processes on different ring communication paths are independent and executed in parallel.

[0210] In other implementations, the data to be standardized is divided into multiple data fragments, including dividing the data to be standardized into multiple data fragments based on the number of at least two ring communication paths.

[0211] The number of ring communication paths refers to the number of ring communication paths that have been determined and can be used to perform full protocol communication. Dividing the data to be regulated based on the number of ring communication paths means that when determining data fragments, the number of ring communication paths is used as the basis for partitioning, so that the partitioned data fragments can correspond to at least two ring communication paths.

[0212] In some implementations, the data to be reduced can be divided into multiple data fragments, the same number as the number of ring communication paths. For example, if two ring communication paths are determined, the data to be reduced can be divided into two data fragments; if seven ring communication paths are determined, the data to be reduced can be divided into seven data fragments. Each data fragment can correspond to one ring communication path, so that each ring communication path processes its corresponding data fragment.

[0213] When the communication capabilities of multiple ring communication paths are similar, the data to be reduced can be evenly divided according to the number of ring communication paths, ensuring that the data size of each data fragment is the same or nearly the same. For example, if the length of the data to be reduced is L and the number of ring communication paths is M, the data to be reduced can be divided into M data fragments, each with a data length of L / M or close to L / M. For cases where the length is not evenly divisible, the data lengths of some data fragments can differ to cover the complete data to be reduced.

[0214] After determining each data fragment, the position of each data fragment within the data to be reduced, as well as the corresponding ring communication path, can be recorded. This record can serve as the basis for data scheduling during subsequent full reduction communication, enabling each ring communication path to acquire its corresponding data fragment and perform full reduction communication on that data fragment.

[0215] In other implementations, the data to be reduced can be first divided into multiple data fragments, and then a corresponding data fragment can be assigned to each ring communication path based on at least two ring communication paths. For example, if a first ring communication path and a second ring communication path exist, the data to be reduced can be divided into a first data fragment and a second data fragment, and a correspondence can be established between the first data fragment and the first ring communication path, as well as between the second data fragment and the second ring communication path.

[0216] It is understandable that by dividing the data to be reduced according to the number of at least two ring communication paths, a clear correspondence can be formed between the data fragments of the data to be reduced and the ring communication paths. Since each ring communication path processes its corresponding data fragment, a single ring communication path does not need to carry all the data to be reduced, thereby reducing the amount of data transmitted by a single ring communication path in a multi-round communication process. Furthermore, it can achieve parallelism in the data dimension and parallelism in the path dimension, thereby improving communication efficiency.

[0217] In some embodiments, performing full protocol communication on data fragments on at least two ring communication paths includes controlling at least two ring communication paths to perform full protocol communication on data fragments in parallel.

[0218] Parallel execution refers to the simultaneous execution of full-protocol communication on at least two ring communication paths, with at least partial overlap in time. It is understood that parallel execution does not require all ring communication paths to start or end at the same time; as long as the communication processes on different ring communication paths occur simultaneously for at least a portion of the time, it can be considered parallel execution.

[0219] During the execution of full-reduction communication, each ring communication path has a corresponding data fragment. For any ring communication path, multiple computing devices participating in that ring communication path can be controlled to transmit their corresponding data fragments along that ring communication path, and reduction operations can be performed on the corresponding data fragments. For another ring communication path, multiple computing devices on that other ring communication path can be controlled to transmit their corresponding data fragments, and reduction operations can be performed on the data fragments.

[0220] Since different ring communication paths correspond to different data fragments, when full reduction communication is executed in parallel on at least two ring communication paths, different data fragments in the data to be reduced can be transmitted and reduced on their respective ring communication paths, without blocking or interfering with each other. Therefore, the data to be reduced does not need to be concentrated on a single ring communication path to complete full reduction communication sequentially. Instead, it can be carried by at least two ring communication paths and processed in at least partially parallel time. This allows the transmission and reduction tasks of the data to be reduced to be distributed across at least two ring communication paths. Compared to the method of carrying all the data to be reduced on a single ring communication path, this reduces the amount of data that a single ring communication path needs to transmit during multiple rounds of communication, thereby reducing the total data transmission time during multiple rounds of communication.

[0221] In some implementations, full-reduction communication is performed on data fragments on at least two ring communication paths, including: on each ring communication path, a reduction distribution phase and a full collection phase are performed on the data fragment corresponding to the ring communication path; the reduction distribution phase and the full collection phase are performed in parallel or interleaved.

[0222] The Reduce-Scatter phase refers to the stage where, along a ring communication path, each computing device performs sending, receiving, and reduction operations on its corresponding data fragments, gradually distributing the reduction results of the data fragments across different computing devices. The Allgather phase, following the Reduce-Scatter phase, refers to the stage where, after the Reduce-Scatter phase, each computing device continues to transmit the reduced data content along the ring communication path, allowing each computing device participating in the ring communication path to obtain the reduction results of its corresponding data fragment.

[0223] For each ring communication path, the protocol distribution phase and the full collection phase are executed independently.

[0224] For each ring communication path, the reduction and distribution phase can be performed on the corresponding data fragment first, followed by the full collection phase. In other words, within a single ring communication path, the reduction and distribution phase and the full collection phase are still executed sequentially according to the processing order of the full reduction communication, to ensure that the corresponding data fragment completes the reduction processing first, and then the reduction result is distributed.

[0225] Parallel execution of the reduction distribution phase and the full collection phase on different ring communication paths means that at least two ring communication paths can execute the corresponding phases within the same or partially overlapping time periods. For example, when the first ring communication path executes the reduction distribution phase of its corresponding data fragment, the second ring communication path can also execute the reduction distribution phase of its corresponding data fragment; when the first ring communication path executes the full collection phase, the second ring communication path can also execute the full collection phase.

[0226] The staggered execution of the reduction distribution phase and the full collection phase on different ring communication paths means that during the parallel advancement of at least two ring communication paths, the phase progress between different ring communication paths does not need to be completely synchronized. For example, the first ring communication path may be executing the reduction distribution phase, while the second ring communication path may have already entered the full collection phase; or, different ring communication paths may advance their respective communication rounds according to their own data fragmentation processing progress, without the need to set a unified phase wait between different ring communication paths.

[0227] In other embodiments, the reduction distribution phase and the full collection phase within each ring communication path are executed sequentially, while the reduction distribution phase and the full collection phase between different ring communication paths are executed in parallel or interleaved.

[0228] In some implementations, the stage state of each ring communication path can be maintained separately. The stage state is used to indicate whether the corresponding ring communication path is currently in the reduction distribution stage, the full collection stage, or the completed state. Based on the stage state of each ring communication path, data transmission, data reception, and reduction operation on each ring communication path can be promoted separately, so that different ring communication paths can be executed in parallel or interleaved.

[0229] Understandably, each ring communication path can still complete the reduction, distribution, and full collection of the corresponding data fragments according to the basic processing order of full-reduction communication. Simultaneously, different ring communication paths do not need to wait sequentially according to a single path order; instead, they can execute the full-reduction communication process of the corresponding data fragments in parallel or interleaved, thereby reducing the limitation of a single ring communication path on the overall completion time of full-reduction communication. Furthermore, at least two ring communication paths can simultaneously carry effective communication loads, enabling parallel reduction and data distribution across multiple ring communication paths, thus improving communication efficiency.

[0230] In some embodiments, another collection communication method is also provided, including the following steps S1 to S10: Step S1: Identify the multiple computing devices participating in the full protocol communication, as well as the physical communication links between the multiple computing devices.

[0231] Step S2: Obtain multiple transmission speeds corresponding to the ring communication path to be constructed, wherein the multiple transmission speeds represent the expected communication rate of the ring communication path to be constructed; traverse the multiple transmission speeds in descending order, and determine the currently traversed transmission speed as the target transmission speed.

[0232] Execute step S3 or step S4 to select path node devices in the ring communication path to be constructed hop by hop: Step S3: For the first device and the second device among multiple computing devices, determine at least two candidate paths between the first device and the second device, and determine the bottleneck bandwidth of each candidate path; the bottleneck bandwidth is the minimum bandwidth of each physical communication link in the candidate path; if the bottleneck bandwidths of each candidate path are different, then determine the target path from at least two candidate paths based on each bottleneck bandwidth; if the bottleneck bandwidths of each candidate path are the same, then determine the hop count of each candidate path, and determine the target path from at least two candidate paths based on the hop count of each candidate path; based on the target path between different computing devices, select the path node device in the ring communication path to be constructed hop by hop from multiple computing devices.

[0233] Step S4: From multiple computing devices, determine the candidate node devices that have not been added to the ring communication path to be constructed, and obtain the topology evaluation parameters of the candidate node devices; based on the topology evaluation parameters, determine the evaluation results of the candidate node devices; according to the evaluation results of each candidate node device, select the path node devices in the ring communication path to be constructed hop by hop.

[0234] Step S5: Deduct the bandwidth usage for this step from the remaining bandwidth of the physical communication links corresponding to the path node devices; if the remaining bandwidth of the physical communication links corresponding to each path node device meets the bandwidth conditions corresponding to the target transmission speed, then construct a ring communication path based on the physical communication links corresponding to each path node device; based on each physical communication link not used by the constructed ring communication path, continue to construct the next ring communication path until at least two ring communication paths are obtained; wherein, each ring communication path includes multiple computing devices, and no two ring communication paths share the same physical communication link.

[0235] Step S6: When the remaining bandwidth of the physical communication link corresponding to the current hop path node device does not meet the bandwidth condition corresponding to the target transmission speed during the construction of the current ring communication path, restore the bandwidth occupancy of the physical communication link corresponding to the path node device that was deducted, and reselect the current hop path node device.

[0236] Step S7: If at least two ring communication paths cannot be constructed for the target transmission speed currently being traversed, return to the step of traversing multiple transmission speeds in descending order until at least two ring communication paths are constructed.

[0237] Step S8: After constructing at least two ring communication paths, determine the total throughput of the at least two ring communication paths; while keeping the total throughput constant, reduce the target transmission speed of the at least two ring communication paths when increasing the number of at least two ring communication paths.

[0238] Step S9: Divide the data to be reduced into multiple data fragments according to the number of at least two ring communication paths; determine the mapping relationship between the multiple data fragments and at least two ring communication paths; and determine the data fragments corresponding to at least two ring communication paths respectively based on the mapping relationship.

[0239] Step S10: On each ring communication path, the data fragments corresponding to the ring communication path are sequentially subjected to the reduction distribution stage and the full collection stage to obtain the full reduction result corresponding to the data to be reduced; the reduction distribution stage and the full collection stage are executed in parallel or interleaved.

[0240] In some embodiments, the performance of the full reduction result after using the above-described collection communication method is shown in Table 1 below: Table 1

[0241] Understandably, experimental tests can be conducted on data of different sizes to be reduced. The test data includes floating-point data (float) ranging from 1MB to 4GB, as well as the corresponding execution time (time), algorithm throughput (algbw), bus bandwidth (busbw), and result correctness (#wrong). Performance metrics under in-place and out-of-place execution modes are recorded respectively.

[0242] Experimental results show that AllReduce significantly improves performance, with bandwidth utilization around 97% (98.1021). Under all test conditions, the data reduction operation completed correctly (#wrong=0), and the algorithm throughput and bus bandwidth increased significantly with increasing data size. For example, with 1 GB of data, the algorithm throughput in out-of-place mode reached approximately 103 GB / s, and the bus bandwidth was approximately 194 GB / s; the throughput and bandwidth performance in in-place mode were similar, indicating that hardware bandwidth resources can be effectively utilized under different execution methods.

[0243] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0244] Furthermore, in this example embodiment, a collection communication device is also provided, with reference to... Figure 11 As shown, the collective communication device 1100 includes: a ring communication path determination module 1110, a data fragmentation determination module 1120, and a full protocol communication module 1130.

[0245] The ring communication path determination module 1110 is used to traverse multiple transmission speeds in descending order, determine the currently traversed transmission speed as the target transmission speed, and determine at least two ring communication paths formed by the physical communication links between multiple computing devices based on the target transmission speed; wherein, the multiple transmission speeds respectively represent the expected communication rate of the ring communication path to be constructed, each ring communication path includes multiple computing devices, and no two ring communication paths share the same physical communication link.

[0246] The data fragment determination module 1120 is used to determine the data fragments in the data to be standardized that correspond to at least two ring communication paths respectively.

[0247] The full reduction communication module 1130 is used to perform full reduction communication on data fragments on at least two ring communication paths to obtain the full reduction result corresponding to the data to be reduced.

[0248] In some example embodiments of this disclosure, based on the foregoing scheme, the aforementioned ring communication path determination module 1110 is configured to: determine multiple computing devices participating in full protocol communication, and the physical communication links between the multiple computing devices; and construct at least two ring communication paths based on the physical communication links between the multiple computing devices.

[0249] In some example embodiments of this disclosure, based on the foregoing scheme, the ring communication path determination module 1110 is configured to: obtain multiple transmission speeds corresponding to the ring communication path to be constructed, wherein the multiple transmission speeds respectively characterize the expected communication rate of the ring communication path to be constructed; traverse the multiple transmission speeds in descending order and determine the currently traversed transmission speed as the target transmission speed; and construct at least two ring communication paths based on the target transmission speed and the physical communication links between the multiple computing devices.

[0250] In some example embodiments of this disclosure, based on the foregoing scheme, the aforementioned ring communication path determination module 1110 is configured to: select path node devices in the ring communication path to be constructed hop by hop from multiple computing devices, and deduct the bandwidth occupancy amount for this time from the remaining bandwidth of the physical communication link corresponding to the path node device; if the remaining bandwidth of the physical communication link corresponding to each path node device meets the bandwidth condition corresponding to the target transmission speed, then construct a ring communication path based on the physical communication link corresponding to each path node device; and continue to construct the next ring communication path based on each physical communication link not used by the constructed ring communication path, until at least two ring communication paths are obtained.

[0251] In some example embodiments of this disclosure, based on the foregoing scheme, the above-mentioned ring communication path determination module 1110 is configured to: during the construction of the current ring communication path, if the remaining bandwidth of the physical communication link corresponding to the path node device of the previous hop does not meet the bandwidth condition corresponding to the target transmission speed, then restore the bandwidth occupancy of the physical communication link corresponding to the path node device that was deducted, and reselect the path node device of the current hop.

[0252] In some example embodiments of this disclosure, based on the foregoing scheme, the aforementioned ring communication path determination module 1110 is configured to: determine candidate node devices that have not been added to the ring communication path to be constructed from multiple computing devices, and obtain topology evaluation parameters of the candidate node devices; determine the evaluation results of the candidate node devices based on the topology evaluation parameters; and select path node devices in the ring communication path to be constructed hop by hop according to the evaluation results of each candidate node device.

[0253] In some example embodiments of this disclosure, based on the foregoing scheme, the ring communication path determination module 1110 is configured to: determine the target path between different computing devices from multiple computing devices; and select path node devices in the ring communication path to be constructed hop by hop from multiple computing devices based on the target path between different computing devices.

[0254] In some example embodiments of this disclosure, based on the foregoing scheme, the aforementioned ring communication path determination module 1110 is configured to: determine at least two candidate paths between the first device and the second device among a plurality of computing devices, and determine the bottleneck bandwidth of each candidate path; the bottleneck bandwidth is the minimum bandwidth of each physical communication link in the candidate path; if the bottleneck bandwidths of each candidate path are different, then determine the target path from the at least two candidate paths based on each bottleneck bandwidth; if the bottleneck bandwidths of each candidate path are the same, then determine the hop count of each candidate path, and determine the target path from the at least two candidate paths based on the hop count of each candidate path.

[0255] In some example embodiments of this disclosure, based on the foregoing scheme, the aforementioned ring communication path determination module 1110 is configured to: if at least two ring communication paths cannot be constructed for the currently traversed target transmission speed, return to the step of traversing multiple transmission speeds in descending order until at least two ring communication paths are constructed.

[0256] In some example embodiments of this disclosure, based on the foregoing scheme, the ring communication path determination module 1110 is configured to: after constructing at least two ring communication paths, determine the total throughput of at least two ring communication paths; and while keeping the total throughput constant, reduce the target transmission speed of at least two ring communication paths when the number of at least two ring communication paths is increased.

[0257] In some example embodiments of this disclosure, based on the foregoing scheme, the data fragmentation determination module 1120 is configured to: divide the data to be reduced into multiple data fragments; determine the mapping relationship between the multiple data fragments and at least two ring communication paths; and determine the data fragments corresponding to at least two ring communication paths based on the mapping relationship.

[0258] In some example embodiments of this disclosure, based on the aforementioned scheme, the data fragmentation determination module 1120 is configured to divide the data to be standardized into multiple data fragments according to the number of at least two ring communication paths.

[0259] In some example embodiments of this disclosure, based on the foregoing scheme, the above-mentioned full protocol communication module 1130 is configured to control at least two ring communication paths to perform full protocol communication in parallel for data fragmentation.

[0260] In some example embodiments of this disclosure, based on the foregoing scheme, the above-mentioned full-reduction communication module 1130 is configured to: on each ring communication path, sequentially execute the reduction distribution stage and the full collection stage on the data fragments corresponding to the ring communication path; and control the reduction distribution stage and the full collection stage on different ring communication paths to be executed in parallel or interleaved.

[0261] The specific details of each module of the above-mentioned collective communication device have been described in detail in the corresponding collective communication method, so they will not be repeated here.

[0262] It should be noted that although several modules or units of the communication device have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0263] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described collection communication method is also provided.

[0264] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be embodied in the following forms: a completely hardware embodiment, a completely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0265] The following reference Figure 12 To describe an electronic device 1200 according to such an embodiment of the present disclosure. Figure 12 The electronic device 1200 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0266] like Figure 12As shown, the electronic device 1200 is presented in the form of a general-purpose computing device. The components of the electronic device 1200 may include, but are not limited to: at least one processing unit 1210, at least one storage unit 1220, a bus 1230 connecting different system components (including storage unit 1220 and processing unit 1210), and a display unit 1240.

[0267] The storage unit stores program code that can be executed by the processing unit 1210, causing the processing unit 1210 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1210 can perform actions such as... Figure 1 In step S110, multiple transmission speeds are traversed in descending order, and the currently traversed transmission speed is determined as the target transmission speed. Based on the target transmission speed, at least two ring communication paths formed by the physical communication links between multiple computing devices are determined. Here, the multiple transmission speeds represent the expected communication rate of the ring communication path to be constructed. Each ring communication path includes multiple computing devices, and no two ring communication paths share the same physical communication link. In step S120, data fragments in the data to be reduced are determined, each corresponding to one of the at least two ring communication paths. In step S130, full reduction communication is performed on the data fragments on the at least two ring communication paths to obtain the full reduction result corresponding to the data to be reduced.

[0268] Storage unit 1220 may include readable media in the form of volatile storage units, such as random access memory (RAM) 1221 and / or cache storage unit 1222, and may further include read-only memory (ROM) 1223.

[0269] Storage unit 1220 may also include a program / utility 1224 having a set (at least one) program module 1225, such program module 1225 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0270] Bus 1230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0271] Electronic device 1200 can also communicate with one or more external devices 1270 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 1200, and / or with any device that enables electronic device 1200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1250. Furthermore, electronic device 1200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1260. As shown, network adapter 1260 communicates with other modules of electronic device 1200 via bus 1230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, Redundant Array of Independent Disks (RAID) systems, tape drives, and data backup storage systems.

[0272] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0273] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0274] refer to Figure 13As shown, a program product 1300 for implementing the above-described collection communication method according to an embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0275] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0276] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0277] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, radio frequency (RF) and so on, or any suitable combination thereof.

[0278] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0279] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0280] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0281] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A collective communication method, characterized in that, The method includes: The system iterates through multiple transmission speeds in descending order, determines the currently traversed transmission speed as the target transmission speed, and, based on the target transmission speed, determines at least two ring communication paths formed by the physical communication links between multiple computing devices. The multiple transmission speeds represent the expected communication rate of the ring communication path to be constructed, each ring communication path includes the multiple computing devices, and no two ring communication paths share the same physical communication link. Determine the data fragments in the data to be standardized that correspond to the at least two ring communication paths respectively; On the at least two ring communication paths, full reduction communication is performed on the data fragments respectively to obtain the full reduction result corresponding to the data to be reduced.

2. The collective communication method according to claim 1, characterized in that, The determination of at least two ring communication paths formed by the physical communication links between multiple computing devices based on the target transmission speed includes: Identify the plurality of computing devices participating in full protocol communication, and the physical communication links between the plurality of computing devices; Based on the target transmission speed and the physical communication links between the multiple computing devices, at least two ring communication paths are constructed.

3. The collective communication method according to claim 2, characterized in that, Based on the target transmission speed and the physical communication links between the multiple computing devices, the construction of the at least two ring communication paths includes: The path node device in the ring communication path to be constructed is selected hop by hop from the plurality of computing devices, and the bandwidth usage for this time is deducted from the remaining bandwidth of the physical communication link corresponding to the path node device. If the remaining bandwidth of the physical communication links corresponding to each path node device meets the bandwidth condition corresponding to the target transmission speed, then a ring communication path is constructed based on the physical communication links corresponding to each path node device. Based on the physical communication links that are not used by the constructed ring communication paths, the next ring communication path is constructed until the at least two ring communication paths are obtained.

4. The collective communication method according to claim 3, characterized in that, The method further includes: During the construction of the current ring communication path, if the remaining bandwidth of the physical communication link corresponding to the path node device of the current hop does not meet the bandwidth condition corresponding to the target transmission speed, the bandwidth occupancy of the physical communication link corresponding to the path node device that was deducted is restored, and the path node device of the current hop is reselected.

5. The collective communication method according to claim 3, characterized in that, The step of selecting path node devices in the ring communication path to be constructed from the plurality of computing devices hop-by-hop includes: From the plurality of computing devices, candidate node devices that have not been added to the ring communication path to be constructed are identified, and the topology evaluation parameters of the candidate node devices are obtained. Based on the topology evaluation parameters, the evaluation results of the candidate node devices are determined; Based on the evaluation results of each candidate node device, the path node devices in the ring communication path to be constructed are selected hop by hop.

6. The collective communication method according to claim 3, characterized in that, The method further includes: Determine the target path between the various computing devices; The step of selecting path node devices in the ring communication path to be constructed from the plurality of computing devices hop-by-hop includes: Based on the target path between different computing devices, path node devices in the ring communication path to be constructed are selected hop-by-hop from the multiple computing devices.

7. The collective communication method according to claim 6, characterized in that, Determining the target path between different computing devices from the plurality of computing devices includes: For the first device and the second device among the plurality of computing devices, at least two candidate paths are determined between the first device and the second device, and the bottleneck bandwidth of each candidate path is determined; the bottleneck bandwidth is the minimum bandwidth of each physical communication link in the candidate path; If the bottleneck bandwidths of each candidate path are different, the target path is determined from the at least two candidate paths based on the bottleneck bandwidths of each path. If the bottleneck bandwidth of each candidate path is the same, the hop count of each candidate path is determined, and the target path is determined from the at least two candidate paths based on the hop count of each candidate path.

8. The collective communication method according to claim 1, characterized in that, The method further includes: If at least two ring communication paths cannot be constructed for the target transmission speed currently being traversed, then return to the step of traversing multiple transmission speeds in descending order until at least two ring communication paths are constructed.

9. The collective communication method according to claim 1, characterized in that, The method further includes: After constructing the at least two ring communication paths, determine the total throughput of the at least two ring communication paths; While keeping the total throughput constant, when the number of the at least two ring communication paths is increased, the target transmission speed of the at least two ring communication paths is reduced.

10. The collective communication method according to any one of claims 1 to 9, characterized in that, The determination of data fragments in the data to be standardized that correspond to the at least two ring communication paths includes: The data to be reduced is divided into multiple data fragments; Determine the mapping relationship between the plurality of data fragments and the at least two ring communication paths; Based on the mapping relationship, data fragments corresponding to the at least two ring communication paths are determined.

11. The collective communication method according to claim 10, characterized in that, The step of dividing the data to be reduced into multiple data fragments includes: Based on the number of the at least two ring communication paths, the data to be standardized is divided into multiple data fragments.

12. The collective communication method according to any one of claims 1 to 9, characterized in that, The step involves performing full reduction communication on the data fragments on the at least two ring communication paths to obtain the full reduction result corresponding to the data to be reduced, including: On each of the ring communication paths, the reduction distribution stage and the full collection stage of the full reduction communication are executed on the data fragments corresponding to the ring communication path to obtain the full reduction result corresponding to the data to be reduced; the reduction distribution stage and the full collection stage are executed in parallel or interleaved.

13. A collective communication device, characterized in that, The device includes: A ring communication path determination module is used to traverse multiple transmission speeds in descending order, determine the currently traversed transmission speed as the target transmission speed, and determine at least two ring communication paths formed by physical communication links between multiple computing devices based on the target transmission speed; wherein, the multiple transmission speeds respectively represent the expected communication rate of the ring communication path to be constructed, each ring communication path includes the multiple computing devices, and no two ring communication paths share the same physical communication link. A data fragmentation determination module is used to determine the data fragments in the data to be reduced that correspond to the at least two ring communication paths respectively; The full reduction communication module is used to perform full reduction communication on the data fragments on the at least two ring communication paths respectively, so as to obtain the full reduction result corresponding to the data to be reduced.

14. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores computer-readable instructions that, when executed by the processor, implement the collective communication method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the collection communication method as described in any one of claims 1 to 12.

Images