Method, device, electronic equipment and storage medium for distributed communication

Abstract

Method, device, electronic equipment and storage medium for distributed communication. The method is applied to a distributed computing system comprising a plurality of computing units, wherein the plurality of computing units are communicatively connected, and the method comprises: dividing communications among the plurality of computing units into a plurality of levels based on sizes of communication bandwidths among the computing units; for each of the plurality of levels: constructing a communication topology of the level based on hardware interconnections of the computing units within the level; and performing a communication process of the level based on the constructed communication topology; wherein the communication processes of the plurality of levels are performed in parallel.

Description

Technical Field

[0001] Embodiments of this disclosure relate to methods, apparatus, electronic devices, and storage media for distributed communication. Background Technology

[0002] Large-scale model competitions have entered the era of trillions of parameters, and distributed training and inference have shifted from "optional" to "mandatory." However, when the scale of distributed clusters expands to the tens of thousands of GPUs, the system bottleneck quickly shifts from computing power to communication efficiency. In many large-scale distributed training or inference tasks, communication overhead accounts for more than 60%—meaning that computing units (such as neural network processing units (NPUs), general-purpose graphics processing units (GPGPUs), or tensor processing units (TPUs)) are not spending most of their time computing, but waiting for data.

[0003] The performance of the communication library has become a key bottleneck in determining the actual computing power release of the Wanka cluster, and it also directly defines the performance ceiling of the entire system.

[0004] Current communication schemes in large-scale distributed clusters have limitations. These schemes treat all computing units in the cluster as peer devices, and in constructing the communication topology, they do not consider the heterogeneous architecture of network infrastructure within and between server nodes and supernodes, thus failing to fully utilize communication bandwidth. Summary of the Invention

[0005] At least one embodiment of this disclosure provides a method, apparatus, electronic device, and storage medium for distributed communication to address the above-mentioned problems.

[0006] According to at least one embodiment of this disclosure, a method for distributed communication is provided, applied to a distributed computing system including multiple computing units, wherein the multiple computing units are communicatively connected. The method includes: dividing the communication between the multiple computing units into multiple layers based on the size of the communication bandwidth between the computing units; for each of the multiple layers: constructing a communication topology for the layer based on the hardware interconnection of the computing units within the layer; and executing a communication process for the layer based on the constructed communication topology; wherein the communication processes of the multiple layers are executed in parallel.

[0007] According to at least one embodiment of this disclosure, the communication bandwidth between computing units in the same level of communication topology is the same.

[0008] According to at least one embodiment of the present disclosure, constructing a hierarchical communication topology based on hardware interconnections of computing units within a hierarchy includes: constructing a hierarchical communication topology based on at least one of hardware interconnections between computing units within a hierarchy, hardware interconnections between computing units within a hierarchy and network communication devices, or hardware interconnections between network communication devices within a hierarchy.

[0009] According to at least one embodiment of this disclosure, the network communication device includes at least one of a network interface card (NIC) or a switch.

[0010] According to at least one embodiment of the present disclosure, the distributed computing system includes multiple computing nodes, each computing node includes at least one computing unit, multiple layers include a first layer, the first layer corresponds to the communication layer within the computing node, and the communication topology of the first layer is used for communication between at least one computing unit within the computing node.

[0011] According to at least one embodiment of the present disclosure, the plurality of layers includes a second layer, the second layer corresponding to an inter-computing node communication layer, and the communication topology of the second layer is used for communication between the plurality of computing nodes.

[0012] According to at least one embodiment of the present disclosure, each computing node includes a plurality of sequentially numbered computing units, and in a second-level communication topology, each of the sequentially numbered computing units of each computing node directly communicates with a computing unit with the same number among the sequentially numbered computing units of another computing node.

[0013] According to at least one embodiment of this disclosure, the communication topology includes a ring communication topology or a tree communication topology.

[0014] According to at least one embodiment of the present disclosure, distributed communication is used for training or inference of a distributed model; and the distributed model includes a neural network model.

[0015] According to at least one embodiment of the present disclosure, the computing unit includes a neural network processing unit, a general graphics processing unit, or a tensor processing unit.

[0016] According to at least one embodiment of this disclosure, an apparatus for distributed communication is provided, the apparatus comprising: a partitioning module configured to partition a plurality of computing units into a plurality of layers based on the communication bandwidth between computing units in a distributed computing system; a construction module configured to construct a communication topology for each of the plurality of layers based on the hardware interconnection of computing units within the layer; and a communication module configured to execute a communication process for each of the plurality of layers based on the constructed communication topology; wherein the communication processes of the plurality of layers are executed in parallel.

[0017] According to at least one embodiment of the present disclosure, an electronic device for distributed communication is provided, the electronic device comprising: one or more processors; and a memory storing one or more computer programs; wherein, when executed by the one or more processors, the one or more computer programs cause the one or more processors to implement the method according to any one of the at least one embodiment of the present disclosure.

[0018] According to at least one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided having computer-executable instructions stored thereon, which, when executed by a processor, cause at least one processor to implement the method according to any one of the at least one embodiments of the present disclosure.

[0019] The method, apparatus, electronic device, and storage medium for distributed communication according to at least one embodiment of the present disclosure can simplify the structure of the communication topology, reduce latency, and improve bandwidth utilization. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0021] Figure 1 A schematic structural diagram of a general-purpose graphics processor (GPGPU) according to at least one embodiment of the present disclosure is shown;

[0022] Figure 2 A method for distributed communication according to at least one embodiment of the present disclosure is shown;

[0023] Figure 3 A network architecture of a distributed computing system having three communication layers according to at least one embodiment of the present disclosure is shown;

[0024] Figure 4 An example of executing a collection communication algorithm based on a distributed computing system having two communication layers, according to at least one embodiment of the present disclosure, is shown;

[0025] Figure 5 An apparatus for distributed communication according to at least one embodiment of the present disclosure is shown;

[0026] Figure 6 An electronic device according to at least one embodiment of the present disclosure is shown;

[0027] Figure 7 A non-transitory computer-readable storage medium according to at least one embodiment of the present disclosure is shown. Detailed Implementation

[0028] The following description, with reference to the accompanying drawings, is provided to aid in a thorough understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. This description includes various specific details to aid understanding but should be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0029] The terms and wording used in the following description and claims are not limited to their dictionary meanings, but are merely intended by the discloser to enable a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of this disclosure is for illustrative purposes only and not for limiting the purpose of this disclosure as defined in the appended claims and their equivalents.

[0030] It should be understood that the singular forms of “one,” “an,” and “the” include plural references unless the context clearly indicates otherwise. Thus, for example, the reference to “component surface” includes one or more such surfaces.

[0031] The terms “comprising” or “may include” refer to the presence of a corresponding disclosed function, operation, or component that may be used in the various embodiments of this disclosure, rather than limiting the presence of one or more additional functions, operations, or features. Furthermore, the terms “comprising” or “having” may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof, but should not be construed as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof.

[0032] The term "or" as used in the various embodiments of this disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B.

[0033] Unless otherwise defined, all terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by one of those skilled in the art as described herein. Common terms as defined in dictionaries are to be interpreted as having a meaning consistent with the context in the relevant technical field and should not be interpreted ideally or overly formally unless expressly defined in this disclosure.

[0034] Throughout the embodiments described herein, the same reference numerals consistently denote the same elements. In this respect, the embodiments may take different forms and should not be construed as limited to the description set forth herein. Therefore, the embodiments are described below only with reference to the accompanying drawings to explain various aspects. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one of..." modify the entire list of elements when preceding it, but do not modify individual elements in the list. In the drawings, the same reference numerals denote the same elements, and for clarity and ease of explanation, the dimensions of components in the drawings may be exaggerated.

[0035] While terms such as "first" and "second" can be used to describe various components, these terms are only used to distinguish one component from another. These terms are not intended to specify that the components are made of different materials or have different structures.

[0036] A singular expression may encompass a plural expression unless it has a distinctly different meaning in the context. When a part "includes" a component, it may further include another component, rather than excluding the other component, unless otherwise stated.

[0037] Furthermore, the terms "part," "module," etc., refer to a unit that processes at least one function or operation, and can be implemented by hardware, software, or a combination thereof.

[0038] With the rapid development of artificial intelligence technology, the race for large-scale models has entered the era of trillions of parameters. In this context, a single computing device can no longer meet the training and inference needs of models, making distributed training technology a necessity rather than an "optional" choice. However, when the scale of computing clusters expands to tens of thousands of computing units (such as Neural Processing Units (NPUs), General Purpose Graphics Processing Units (GPGPUs), or Tensor Processing Units (TPUs), the system's performance bottleneck quickly shifts from computing power to communication efficiency. In many large-scale training tasks, communication overhead accounts for more than 60% of the total time, meaning that computing devices spend most of their time idle rather than performing effective computations. Therefore, the performance of the communication library has become a key bottleneck determining whether the true computing power of a multi-core cluster can be fully released, directly defining the performance ceiling of the entire distributed computing system.

[0039] In current communication schemes, all computing units in a distributed computing system are treated as peer devices. During the construction of communication topologies (e.g., ring or tree topologies), the multi-level heterogeneous architecture of network infrastructure within and between server nodes and supernodes is not considered. This results in the simultaneous existence of intra-server and intra-supernode communication, as well as inter-server and inter-supernode communication within the same topology. In other words, all intra-node and inter-node communication channels are treated as the same, failing to differentiate between different types of channels during topology construction. Since different types of communication have significantly different bandwidths and latency, the time consumed by each operation in the communication process is limited by the slowest channel in the entire communication link. This creates a significant "barrel effect," where high-bandwidth communication channels in the topology have to wait for low-bandwidth channels. Therefore, the communication bandwidth of the distributed computing system cannot be fully utilized, its performance potential is wasted, and this severely restricts the overall communication efficiency of large-scale distributed computing systems.

[0040] Meanwhile, in the current communication scheme, the communication topology is complex due to the construction of communication topology across network facilities with multi-level heterogeneous architecture. The communication operation path is long and there is a risk of cross-switch (e.g., peripheral component interconnection fast (PCIe) switch), resulting in high communication latency, which also seriously affects the overall communication efficiency of distributed computing system.

[0041] At least one embodiment of this disclosure provides a method, apparatus, electronic device, and storage medium for distributed communication. For example, at least one embodiment introduces topology-aware heterogeneous communication scheduling to dynamically match communication paths with hardware capabilities, achieving precise alignment of load and bandwidth, and releasing the true performance of the cluster; for example, at least one embodiment can simplify the structure of the communication topology, shorten the operation path, improve the utilization of communication bandwidth, and achieve more flexible network configuration.

[0042] The following describes in detail, with reference to the accompanying drawings, a method, apparatus, electronic device, and storage medium for distributed communication according to at least one embodiment of the present disclosure.

[0043] Figure 1 A schematic structural diagram of a general-purpose graphics processor (GPGPU) according to at least one embodiment of the present disclosure is shown.

[0044] like Figure 1 As shown, a general-purpose graphics processor is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 1The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0045] like Figure 1 As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 1 The system is structured as Computation Unit 1, Computation Unit 2, ..., Computation Unit N, where N is a positive integer. Each Computation Unit (CU) performs arithmetic and logical operations, such as accumulation, reduction, and standard addition, subtraction, multiplication, and division. A Computation Unit includes multiple cores (also called computational kernels), each of which includes an Arithmetic Logic Unit (ALU), a floating-point unit, etc. These cores are used to execute specific computational tasks. Furthermore, the Computation Unit also includes registers (e.g., ...). Figure 1 The register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0046] like Figure 1 As shown, each computing unit also provides a tensor core for performing tensor-related computations, such as tensor shrinking operations. Tensor cores can accelerate tensor operations such as matrix multiplication. Tensor cores in multiple computing units can be scheduled and controlled uniformly.

[0047] like Figure 1 As shown, each streaming processor cluster also provides a buffer for caching data across the N computing units within the streaming processor cluster.

[0048] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1 (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0049] In each computing unit, the thread beam scheduling / distribution module ( Figure 1 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can execute concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory-executed instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory (e.g., [example cache]). Figure 1 High Bandwidth Memory (HBM) is used for read and write operations.

[0050] Figure 2 A method 200 for distributed communication according to at least one embodiment of the present disclosure is shown. For example... Figure 2 As shown, method 200 includes steps 202-206.

[0051] According to at least one embodiment of this disclosure, the method 200 for distributed communication can be applied to a distributed computing system comprising multiple computing units. For example, according to at least one embodiment of this disclosure, the multiple computing units may include one or more neural network processing units (NPUs), general-purpose graphics processing units (GPGPUs), or tensor processing units (TPUs), etc., but the embodiments of this disclosure are not limited thereto. The distributed computing system according to at least one embodiment of this disclosure can be a distributed cluster at the tens of thousands of computing units level; for example, the distributed computing system according to at least one embodiment of this disclosure can be a distributed cluster comprising tens of thousands of computing units.

[0052] According to at least one embodiment of this disclosure, a distributed computing system comprising multiple computing units can be used for training or inference of a distributed model. For example, distributed communication between the multiple computing units included in the distributed computing system can be used for training or inference of the distributed model. The distributed model can be a deep learning model with a huge number of parameters, a massive amount of training data, and strong learning and generalization capabilities. Distributed models are typically based on architectures such as Transformers, capable of learning general knowledge from massive amounts of data and performing well in various downstream tasks, even possessing certain inference, generation, and zero-shot / few-shot learning capabilities. For example, according to at least one embodiment of this disclosure, the distributed model can include a large-scale neural network model, but the embodiments of this disclosure are not limited thereto.

[0053] According to at least one embodiment of this disclosure, multiple computing units of a distributed computing system can be communicatively connected. For example, according to at least one embodiment of this disclosure, multiple computing units of a distributed computing system can be communicatively connected through various communication protocols. According to at least one embodiment of this disclosure, the various communication protocols include, but are not limited to, the following: For example, P2Plink, which can be used for communication connections between multiple computing units (e.g., a graphics processing unit (GPU)) within the same node, is a high-speed interconnect technology with extremely high bandwidth and extremely low latency; PCIe, which can be used for communication connections between different hardware components within a node or between nodes, for example, PCIe can be used for communication connections between hardware such as a central processing unit (CPU), GPU, and network interface card within a server node, where PCIe has lower bandwidth and higher latency than P2Plink; or High-speed Ethernet or InfiniBand, which can be used for communication connections between nodes, with bandwidth further lower than PCIe and latency further higher than PCIe. Because high-speed Ethernet or InfiniBand used for inter-node communication has lower bandwidth and higher latency compared to P2Plink and PCIe, it may become the slowest link in a distributed computing system and the bottleneck of communication in the distributed computing system.

[0054] refer to Figure 2 For the distributed communication method 200, in step 202, the communication between multiple computing units can be divided into multiple levels based on the size of the communication bandwidth between computing units.

[0055] For example, according to at least one example of this disclosure, the communication connections between computing units in a distributed computing system can have different communication bandwidths. For example, the communication connections between multiple computing units can include high-bandwidth connections, medium-bandwidth connections, and low-bandwidth connections; however, this is merely an example, and according to at least one embodiment of this disclosure, the communication connections between multiple computing units can include more or fewer layers.

[0056] According to at least one embodiment of the present disclosure, for example, a high-bandwidth connection may include a communication connection via P2Plink, a medium-bandwidth connection may include a communication connection via PCIe, and a low-bandwidth connection may include a communication connection via high-speed Ethernet or InfiniBand. However, this is merely an example, and the high-bandwidth connection, medium-bandwidth connection, and low-bandwidth connection according to at least one embodiment of the present disclosure may each include various different types of communication connections.

[0057] According to at least one embodiment of this disclosure, communication between multiple computing units in a distributed computing system can be divided into multiple levels based on the communication bandwidth between the computing units. For example, according to at least one embodiment of this disclosure, the communication connection between multiple computing units may include high-bandwidth connections, medium-bandwidth connections, and low-bandwidth connections. The communication between multiple computing units in a distributed computing system can be divided into multiple levels based on the communication bandwidth of these different communication connections, such as a high-bandwidth communication level, a medium-bandwidth communication level, and a low-bandwidth communication level. For example, the communication between multiple computing units can be divided into a P2Plink communication level, a PCIe communication level, and a high-speed Ethernet or InfiniBand communication level.

[0058] like Figure 2 As shown, in step 204, the communication topology of a layer can be constructed based on the hardware interconnection of computing units within the layer for each of the multiple layers.

[0059] According to at least one embodiment of this disclosure, after dividing the communication between multiple computing units into multiple layers based on the communication bandwidth of the computing units, a communication topology can be constructed for each layer based on the hardware interconnection of the computing units within that layer. The constructed communication topology can be used for communication between computing units within that layer. For example, according to at least one embodiment of this disclosure, after dividing the communication between multiple computing units into a high-bandwidth communication layer, a medium-bandwidth communication layer, and a low-bandwidth communication layer, communication topologies for the high-bandwidth communication layer, the medium-bandwidth communication layer, and the low-bandwidth communication layer can be constructed based on the hardware interconnection of the computing units within the high-bandwidth communication layer, the medium-bandwidth communication layer, and the low-bandwidth communication layer, respectively.

[0060] In step 206, method 200 may include performing a communication process for each of the multiple layers based on the constructed communication topology.

[0061] According to at least one embodiment of this disclosure, communication processes at multiple levels can be executed in parallel. For example, according to at least one embodiment of this disclosure, after constructing communication topologies for a high-bandwidth communication layer, a medium-bandwidth communication layer, and a low-bandwidth communication layer, respectively, communication processes at the high-bandwidth, medium-bandwidth, and low-bandwidth communication layers can be executed based on these topologies. According to at least one embodiment of this disclosure, communication processes at the high-bandwidth, medium-bandwidth, and low-bandwidth communication layers can be executed in parallel.

[0062] According to at least one embodiment of the present disclosure Figure 2The method for distributed communication shown divides communication between computing units into multiple layers based on communication bandwidth, constructs a communication topology based on hardware interconnection within each layer, and executes the communication process of multiple layers of communication topology in parallel. This simplifies the structure of the communication topology at each layer, shortens the operation path, completes data communication on all computing units with fewer operation steps, improves the utilization of communication bandwidth, and enables more flexible network configuration.

[0063] According to at least one embodiment of this disclosure, the method for distributed communication may further include dividing communication between multiple computing units into multiple layers, such that computing units communicate via the same communication connection within each layer. For example, according to at least one embodiment of this disclosure, communication between multiple computing units may be divided into a high-bandwidth communication layer, a medium-bandwidth communication layer, and a low-bandwidth communication layer, and the same communication connection may be used for communication within each of the three layers. For example, according to at least one embodiment of this disclosure, the high-bandwidth communication layer may be a P2Plink communication layer, wherein one or more computing units communicate via P2Plink; the medium-bandwidth communication layer may be a PCIe communication layer, wherein one or more computing units communicate via PCIe; and the low-bandwidth communication layer may be a high-speed Ethernet or InfiniBand communication layer, wherein one or more computing units communicate via high-speed Ethernet or InfiniBand.

[0064] According to at least one embodiment of this disclosure, the communication bandwidth between computing units in the same level of communication topology can be the same. For example, according to at least one embodiment of this disclosure, the communication between multiple computing units can be divided into multiple levels based on the size of the communication bandwidth between computing units, and the hierarchical communication topology can be constructed based on the hardware interconnection of computing units within each level, so that the communication between computing units in the same level of communication topology has the same communication bandwidth. For example, the communication between multiple computing units can be divided into high-bandwidth communication levels, medium-bandwidth communication levels, and low-bandwidth communication levels (e.g., P2Plink communication level, PCIe communication level, and high-speed Ethernet or InfiniBand communication level) based on the size of the communication bandwidth between computing units, and high-bandwidth communication topologies can be constructed based on the hardware interconnection of computing units within the high-bandwidth communication level, medium-bandwidth communication level, and low-bandwidth communication level, respectively, wherein the communication bandwidth between computing units in each level of communication topology is the same.

[0065] According to at least one embodiment of this disclosure, by dividing the communication between multiple computing units into multiple layers and making the communication bandwidth between computing units in the same layer of the communication topology the same, communication with the same bandwidth can be aggregated into the same layer. Thus, in each layer, there will be no situation where a high-bandwidth communication channel needs to wait for a low-bandwidth communication channel, and the bandwidth of each communication channel can be fully utilized to improve communication efficiency.

[0066] According to at least one embodiment of the present disclosure, the constructed communication topology may include a ring communication topology or a tree communication topology, but the embodiments of the present disclosure are not limited thereto. The method for distributed communication according to the embodiments of the present disclosure can construct communication topologies of various structures based on the hardware interconnection of computing units.

[0067] According to at least one embodiment of this disclosure, a distributed computing system includes multiple computing nodes, each of which includes at least one computing unit. For example, according to at least one embodiment of this disclosure, the distributed computing system may include multiple server computing nodes, and each server computing node may include at least one computing unit (e.g., a GPU). However, this is merely an example, and according to at least one embodiment of this disclosure, the distributed computing system may include other types of computing nodes. According to at least one embodiment of this disclosure, multiple layers may include a first layer, which may correspond to an intra-computational communication layer (e.g., an intra-server computing node communication layer), and the communication topology of the first layer is used for communication between at least one computing unit (e.g., at least one GPU within a server computing node) within the computing node. For example, according to at least one embodiment of this disclosure, the first layer may correspond to an intra-computational communication layer communicating via a P2Plink communication connection, and the first layer may correspond to a high-bandwidth communication layer.

[0068] According to at least one embodiment of this disclosure, the multiple layers may further include a second layer, which may correspond to an inter-component communication layer, and the communication topology of the second layer is used for communication between the multiple compute nodes. For example, the second layer may correspond to a server inter-component communication layer, and the communication topology of the second layer is used for communication between the multiple server compute nodes. According to at least one embodiment of this disclosure, for example, the second layer may correspond to an inter-component communication layer communicating via a PCIe communication connection, and the second layer may correspond to a medium-bandwidth communication layer. Alternatively, the second layer may correspond to a communication layer communicating via a high-speed Ethernet or InfiniBand communication connection, and the second layer may correspond to a low-bandwidth communication layer.

[0069] According to at least one embodiment of this disclosure, constructing a hierarchical communication topology based on hardware interconnections of computing units within a hierarchy may include constructing a hierarchical communication topology based on at least one of hardware interconnections between computing units within a hierarchy, hardware interconnections between computing units within a hierarchy and network communication devices, or hardware interconnections between network communication devices within a hierarchy.

[0070] For example, according to at least one embodiment of this disclosure, the network communication device may include at least one of a network interface card (NIC) or a switch. For example, when constructing a communication topology for a communication layer within a computing node, the communication topology can be constructed based on the hardware interconnections between computing units within the communication layer of the computing node.

[0071] For example, when constructing a communication topology for a communication layer between computing nodes, the communication topology can be constructed based on at least one of the following: hardware interconnection between computing units within the layer, hardware interconnection between computing units within the layer and network communication devices within the layer, or hardware interconnection between network communication devices within the layer. For example, the network communication devices may include at least one of network interface cards (NICs) or switches, but the embodiments of this disclosure are not limited thereto.

[0072] According to at least one embodiment of the present disclosure, each computing node (e.g., a server computing node) may include a plurality of sequentially numbered computing units, and in a communication topology of a second level (e.g., an inter-computing node communication level), each of the sequentially numbered computing units of each computing node directly communicates with a computing unit with the same number among the sequentially numbered computing units of another computing node.

[0073] For example, a distributed computing system may include two computing nodes, namely node 1 and node 2, and each computing node may include eight computing units, namely GPU0-GPU7. According to at least one embodiment of this disclosure, in the communication hierarchy between computing nodes 1 and 2, CPU0 of node 1 can communicate directly with GPU0 of node 2, CPU1 of node 1 can communicate directly with GPU1 of node 2, CPU2 of node 1 can communicate directly with GPU2 of node 2, and so on.

[0074] Each computing unit in a sequentially numbered computing unit on each computing node communicates directly with the computing unit with the same number in a sequentially numbered computing unit on another computing node. In the communication at the inter-computing node communication level, each computing unit (e.g., GPU) only needs to access its network card with the same number, avoiding cross-switch situations, thus avoiding latency caused by cross-switch and improving communication efficiency.

[0075] According to at least one embodiment of this disclosure, distributed communication among multiple computing units of a distributed computing system can be used for training or inference of a distributed model. For example, the training task of a deep learning model (e.g., a large-scale neural network model) can be split into multiple computing units, and through collaborative computing and communication, the training process can be accelerated, the model size can be expanded, or larger datasets can be processed. For example, according to at least one embodiment of this disclosure, collective communication (e.g., collective communication such as All-Gather, ReduceScatter, etc.) of a distributed model can be performed through multiple computing units of a distributed computing system.

[0076] Figure 3 A network architecture of a distributed computing system 300 having three communication layers according to at least one embodiment of the present disclosure is shown.

[0077] According to at least one embodiment of this disclosure, communication between multiple computing units in a distributed computing system can be divided into three levels based on the communication bandwidth between them. For example, as Figure 3 As shown, a distributed computing system may include three communication layers: layer 1, layer 2, and layer 3. According to at least one embodiment of this disclosure, layer 1 may correspond to a high-bandwidth communication layer, layer 2 may correspond to a medium-bandwidth communication layer, and layer 3 may correspond to a low-bandwidth communication layer. Figure 3 The distributed computing system with three communication layers shown is merely an example, but the embodiments of this disclosure are not limited thereto. Distributed computing systems according to at least one embodiment of this disclosure may have more or fewer communication layers.

[0078] For example, according to at least one embodiment of this disclosure, level 1 may correspond to the intra-node communication level, level 2 may correspond to the inter-node communication level, and level 3 may correspond to the inter-rack communication level.

[0079] like Figure 3 As shown, a distributed computing system can have four communication layers (level 1), two communication layers (level 2), and one communication layer (level 3). For example... Figure 3 As shown, each level 1 communication layer can include communication between four computing units (e.g., four GPUs). For example, in Figure 3In the illustrated embodiment, the four Level 1 communication layers may respectively include communication between the four computing units identified as A, communication between the four computing units identified as B, communication between the four computing units identified as C, and communication between the four computing units identified as D. According to at least one embodiment of this disclosure, the four computing units in Level 1 can communicate via a high-bandwidth connection, therefore Level 1 may correspond to a high-bandwidth communication layer.

[0080] According to at least one embodiment of this disclosure, such as Figure 3 As shown, the two communication levels 2 can respectively include communication between the four computing units identified as A and the four computing units identified as B, and communication between the four computing units identified as C and the four computing units identified as D. According to at least one embodiment of this disclosure, level 2 can communicate via a medium bandwidth connection, therefore level 2 can correspond to a medium bandwidth communication level.

[0081] According to at least one embodiment of this disclosure, such as Figure 3 As shown, the level 3 communication layer may include communication between eight computing units identified as A and B and eight computing units identified as C and D. According to at least one embodiment of this disclosure, level 3 may communicate via a low-bandwidth connection, and therefore level 3 may correspond to a low-bandwidth communication layer.

[0082] According to at least one embodiment of the present disclosure, for example, a high-bandwidth connection may include a communication connection via P2Plink, a medium-bandwidth connection may include a communication connection via PCIe, and a low-bandwidth connection may include a communication connection via high-speed Ethernet or InfiniBand. However, this is merely an example, and the high-bandwidth connection, medium-bandwidth connection, and low-bandwidth connection according to at least one embodiment of the present disclosure may each include various different types of communication connections.

[0083] According to at least one embodiment of this disclosure, a communication topology for each of layers 1 to 3 can be constructed based on the hardware interconnection of computing units within that layer. According to at least one embodiment of this disclosure, after dividing the communication between multiple computing units into layers 1 to 3 based on the communication bandwidth between computing units, a communication topology for each of layers 1 to 3 can be constructed based on the hardware interconnection of computing units within that layer. The constructed communication topology can be used for communication between computing units within that layer. For example, as... Figure 3 As shown, for level 1, the communication topology of level 1 can be constructed based on the hardware interconnection of computing units within level 1. For example, as... Figure 3As shown, based on the hardware interconnection of the four computing units identified as A, B, C, and D, respectively, the four computing units in each level 1 can be constructed into a ring communication topology. Although Figure 3 The illustration shows a communication topology of level 1 constructed as a ring communication topology. However, the embodiments of this disclosure are not limited thereto. According to at least one embodiment of this disclosure, various types of communication topologies, such as tree topologies, can be constructed based on the hardware interconnection of computing units.

[0084] Similarly, according to at least one embodiment of this disclosure, a communication topology of level 2 or level 3 can be constructed based on the hardware interconnection of computing units in level 2 and the hardware interconnection of computing units in level 3. For example, the communication topology may include a ring topology or a tree topology.

[0085] According to at least one embodiment of this disclosure, the communication bandwidth between computing units in the same level of communication topology can be the same. For example, communication between computing units in a level 1 communication topology can be performed via a high-bandwidth connection with the same communication bandwidth, communication between computing units in a level 2 communication topology can be performed via a medium-bandwidth connection with the same communication bandwidth, and communication between computing units in a level 3 communication topology can be performed via a low-bandwidth connection with the same communication bandwidth.

[0086] According to at least one embodiment of this disclosure, it is possible to target Figure 3 Each of the layers 1 to 3 shown executes a communication process based on the constructed communication topology. According to at least one embodiment of this disclosure, the communication processes of multiple layers can be executed in parallel.

[0087] For example, a level 1 communication process can be executed based on a level 1 communication topology, a level 2 communication process can be executed based on a level 2 communication topology, and a level 3 communication process can be executed based on a level 3 communication topology. For example, in Figure 3In the embodiments described, the communication process at level 1 can be the communication process between four computing units identified as A, four computing units identified as B, four computing units identified as C, and four computing units identified as D, respectively; the communication process at level 2 can be the communication process between four computing units identified as A and four computing units identified as B, and between four computing units identified as C and four computing units identified as D, respectively; and the communication process at level 3 can be the communication process between eight computing units identified as A and B and eight computing units identified as C and D. According to at least one embodiment of this disclosure, the communication processes at level 1, level 2, and level 3 can be executed in parallel.

[0088] According to at least one embodiment of this disclosure, constructing a hierarchical communication topology based on hardware interconnections of computing units within a hierarchy may include constructing the hierarchical communication topology based on at least one of hardware interconnections between computing units within a hierarchy, hardware interconnections between computing units within a hierarchy and network communication devices, or hardware interconnections between network communication devices within a hierarchy. For example, according to at least one embodiment of this disclosure, the network communication device may include at least one of a network interface card (NIC) or a switch.

[0089] For example, according to at least one embodiment of this disclosure, in Figure 3 In the illustrated embodiments, level 1 may correspond to the intra-node communication level, and the communication topology of level 1 can be constructed based on the hardware interconnection between computing units within level 1; level 2 may correspond to the inter-node communication level, and the communication topology of level 2 can be constructed based on at least one of the hardware interconnection between computing units within level 2, the hardware interconnection between computing units within level 2 and network communication devices, or the hardware interconnection between network communication devices within level 2. For example, the network communication device may include a network interface card (NIC); level 3 may correspond to the inter-rack communication level, and the communication topology of level 3 can be constructed based on at least one of the hardware interconnection between computing units within level 3, the hardware interconnection between computing units within level 3 and network communication devices, or the hardware interconnection between network communication devices within level 3. For example, the network communication device may include a NIC or a switch. It should be understood that the embodiments described above are merely examples, and the embodiments of this disclosure are not limited thereto. For example, for the intra-node communication level, the communication topology of the level may also be constructed based on at least one of the hardware interconnection between computing units within the level and network communication devices, or the hardware interconnection between network communication devices within the level.

[0090] According to at least one embodiment of this disclosure, each computing node (e.g., a server computing node) may include a plurality of sequentially numbered computing units, and in a communication topology at the inter-computational node communication level, each of the sequentially numbered computing units of each computing node directly communicates with a computing unit having the same number among the sequentially numbered computing units of another computing node. For example, in Figure 3 In the illustrated embodiment, level 1 can correspond to the communication level within a computing node; for example, level 1 can correspond to communication between four computing units within a computing node. For example, in... Figure 3 In the illustrated embodiment, there may be four computing nodes, and each computing node includes four computing units. Specifically, the first computing node may include four computing units identified as A, the second computing node may include four computing units identified as B, the third computing node may include four computing units identified as C, and the fourth computing node may include four computing units identified as D.

[0091] For example, according to at least one embodiment of this disclosure, the computing units in each of the four levels 1 can be sequentially numbered as having numbers 1, 2, 3 and 4, that is, the four computing units in each of the four computing nodes can be sequentially numbered as having numbers 1, 2, 3 and 4.

[0092] According to at least one embodiment of this disclosure, in Figure 3 In the illustrated embodiment, level 2 can correspond to the inter-node communication level. Figure 3 In the illustrated embodiment, the two levels 2 can respectively correspond to the communication between the first computing node and the second computing node, and the communication between the third computing node and the fourth computing node. According to at least one embodiment of this disclosure, in the communication topology of level 2, each of the four sequentially numbered computing units of one computing node directly communicates with the computing units with the same number among the four sequentially numbered computing units of another computing node.

[0093] For example, in the communication topology of layer 2 corresponding to the communication between the first computing node and the second computing node, the computing unit numbered 1 in the first computing node communicates directly with the computing unit numbered 1 in the second computing node, the computing unit numbered 2 in the first computing node communicates directly with the computing unit numbered 2 in the second computing node, the computing unit numbered 3 in the first computing node communicates directly with the computing unit numbered 3 in the second computing node, and the computing unit numbered 4 in the first computing node communicates directly with the computing unit numbered 4 in the second computing node.

[0094] According to at least one embodiment of this disclosure, distributed communication among multiple computing units of the distributed computing system 300 can be used for training or inference of a distributed model. For example, the training task of a deep learning model (e.g., a large-scale neural network model) can be split across multiple computing units of the distributed computing system 300, accelerating the training process, scaling the model, or processing larger datasets through collaborative computing and communication. For example, according to at least one embodiment of this disclosure, aggregate communication of the distributed model (e.g., aggregate communication such as All-Gather, ReduceScatter, etc.) can be performed through multiple computing units of the distributed computing system 300.

[0095] Figure 4 Example 400 of executing a collection communication algorithm based on a distributed computing system having two communication layers, according to at least one embodiment of the present disclosure, is shown.

[0096] In the training or inference of distributed large models, ensemble communication is the core operation for coordinating data synchronization between multiple computing units (such as GPUs). For example, All-Gather and ReduceScatter are commonly used ensemble communication algorithms.

[0097] Below, regarding Figure 4 The illustrated embodiment, using the All-Gather algorithm as an example, describes Example 400 of executing a set communication algorithm based on a distributed computing system with two communication layers according to at least one embodiment of this disclosure. However, embodiments of this disclosure are not limited thereto. Figure 4 The example 400 shown can be applied to various set communication algorithms.

[0098] refer to Figure 4 A distributed computing system may include two computing nodes—node 1 and node 2. For example, according to at least one embodiment of this disclosure, node 1 and node 2 may be server computing nodes, but the embodiments of this disclosure are not limited thereto. According to at least one embodiment of this disclosure, node 1 and node 2 may each include multiple computing units, for example, in Figure 4 In the illustrated embodiment, node 1 and node 2 may each include multiple GPUs, but this is merely an example. According to at least one embodiment of this disclosure, node 1 and node 2 may include multiple computing units of any type, such as one or more neural network processing units (NPUs), general-purpose graphics processing units (GPGPUs), or artificial intelligence processors such as tensor processing units (TPUs).

[0099] refer to Figure 4 Node 1 and Node 2 each contain 8 GPUs, and the 8 GPUs in each node are sequentially numbered GPU0-GPU7. Furthermore, Figure 4Each GPU in the dataset can have a data block; for example, each GPU can have a gradient data block from the complete gradient data used for distributed training tasks.

[0100] For example, according to at least one embodiment of this disclosure, Figure 4 The distributed computing system in the example can be used for distributed training tasks, which are performed on 16 GPUs across nodes 1 and 2. Each GPU processes a different batch of training data and computes the gradient data for its respective training data. For example, as... Figure 4 As shown, GPUs 0-7 of node 1 can each have gradient data blocks grad0-grad7, while GPUs 0-GPU7 of node 2 can each have gradient data blocks grad8-grad15. In distributed training, to update the model, each GPU needs to obtain the gradient data computed by all GPUs, i.e., grad0-grad15. According to at least one embodiment of this disclosure, the All-Gather algorithm can be used to ensure that all GPUs obtain the complete gradient data grad0-grad15.

[0101] exist Figure 4 In the illustrated embodiment, GPUs within a node can communicate via a high-bandwidth connection, while GPUs between nodes can communicate via a medium-bandwidth connection or a low-bandwidth connection.

[0102] According to at least one embodiment of this disclosure, inter-GPU communication can be divided into intra-node communication layers and inter-node communication layers based on inter-GPU communication bandwidth. For example, refer to... Figure 4 The intra-node communication hierarchy can be communication between GPU0-GPU7 within node 1 and communication between GPU0-GPU7 within node 2 (by...). Figure 4 (represented by solid arrows), and the inter-node communication level can be communication between GPU0-GPU7 of node 1 and GPU0-GPU7 of node 2 (by...). Figure 4 (The dashed arrow in the image indicates this).

[0103] According to at least one embodiment of this disclosure, when... Figure 4 The distributed computing system illustrated, by dividing communication between multiple GPUs into intra-node communication layers and inter-node communication layers, allows the construction of communication topologies for each layer based on the hardware interconnects of the GPUs within that layer. For example, as... Figure 4 As shown, GPUs 0-GPU 7 within node 1 and GPUs 0-GPU 7 within node 2 can each be configured as a ring communication topology. The communication topology between node 1 and node 2 can also be configured as a ring communication topology, and in the communication topology between node 1 and node 2, GPUs with the same number communicate directly with GPUs with the same number.

[0104] According to at least one embodiment of this disclosure, a hierarchical communication topology is constructed based on at least one of hardware interconnection between computing units within a hierarchy, hardware interconnection between computing units within a hierarchy and network communication devices, or hardware interconnection between network communication devices within a hierarchy. For example, in Figure 4 In the illustrated embodiments, for the intra-node communication layer, the communication topology can be constructed based on the hardware interconnections between GPUs within the node. For the inter-node communication layer, the communication topology can be constructed based on the hardware interconnections between GPUs, the hardware interconnections between GPUs and network communication devices, and the hardware interconnections between network communication devices, wherein the network communication devices may include network interface cards (NICs) or switches. However, this is merely an example. According to at least one embodiment of this disclosure, for the intra-node communication layer, the hardware interconnections between GPUs and network communication devices, as well as the hardware interconnections between network communication devices, can also be considered when constructing the communication topology.

[0105] According to at least one embodiment of this disclosure, in Figure 4 In Example 400, after completing the construction of the communication topology of the intra-node communication layer and the inter-node communication layer, the communication process of the intra-node communication layer and the communication process of the inter-node communication layer can be executed based on the constructed communication topology to execute All-Gather communication of the 16 GPUs included in the distributed computing system, that is, to execute gradient data synchronization of the 16 GPUs so that each of the 16 GPUs has complete gradient data grad0-gard15.

[0106] According to at least one embodiment of this disclosure, the communication process at the intra-node communication level and the communication process at the inter-node communication level in Example 400 can be executed in parallel. For example, refer to... Figure 4 According to at least one embodiment of the present disclosure, while GPU0-GPU7 in node 1 are communicating based on the communication topology to perform the synchronization of gradient data grad0-grad7, and GPU0-GPU7 in node 2 are communicating based on the communication topology to perform the synchronization of gradient data grad8-grad15, the communication process between node 1 and node 2 can be executed in parallel to synchronize the gradient data grad0-grad7 in node 1 to node 2 and synchronize the gradient data grad8-grad15 in node 2 to node 1.

[0107] For example, refer to Figure 4The communication process between nodes 1 and 2 can be executed through communication between GPUs with the same ID. For example, GPUs 0-7 in node 1 communicate based on the communication topology to synchronize gradient data grad0-grad7, and GPUs 0-GPU7 in node 2 communicate based on the communication topology to synchronize gradient data grad8-grad15. Simultaneously, GPU 0 in node 1 can communicate with GPU 0 in node 2 to synchronize gradient data grad0 and grad8, GPU 1 in node 1 can communicate with GPU 1 in node 2 to synchronize gradient data grad1 and grad9, GPU 2 in node 1 can communicate with GPU 2 in node 2 to synchronize gradient data grad2 and grad10, and so on. Ultimately, when the intra-node communication layer and the inter-node communication layer are executed in parallel, each of the 16 GPUs in the distributed computing system can have complete gradient data grad0-grad15.

[0108] Figure 5 An apparatus 500 for distributed communication according to at least one embodiment of the present disclosure is shown.

[0109] like Figure 5 As shown, the device 500 may include a partitioning module 502. The partitioning module 502 may be configured to divide multiple computing units into multiple levels based on the communication bandwidth between computing units in the distributed computing system.

[0110] For example, according to at least one example of this disclosure, the communication connections between computing units in a distributed computing system can have different communication bandwidths. For example, the communication connections between multiple computing units can include high-bandwidth connections, medium-bandwidth connections, and low-bandwidth connections; however, this is merely an example, and according to at least one embodiment of this disclosure, the communication connections between multiple computing units can include more or fewer layers.

[0111] According to at least one example of this disclosure, the partitioning module 502 can be configured to divide the communication between multiple computing units into multiple levels based on the size of the communication bandwidth between computing units in a distributed computing system. For example, according to at least one embodiment of this disclosure, the communication connection between multiple computing units may include a high-bandwidth connection, a medium-bandwidth connection, and a low-bandwidth connection. The partitioning module 502 can be configured to divide the communication between multiple computing units in a distributed computing system into multiple levels based on the communication bandwidth of the aforementioned different communication connections, such as a high-bandwidth communication level, a medium-bandwidth communication level, and a low-bandwidth communication level.

[0112] like Figure 5As shown, the device 500 may also include a construction module 504. The construction module 504 may be configured to construct a communication topology for each of the multiple layers based on the hardware interconnection of computing units within the layer.

[0113] According to at least one embodiment of this disclosure, after dividing the communication between multiple computing units into multiple layers based on the communication bandwidth of the computing units, the construction module 504 can be configured to construct a communication topology for each of the multiple layers based on the hardware interconnection of the computing units within that layer. The constructed communication topology can be used for communication between computing units within that layer. For example, according to at least one embodiment of this disclosure, after dividing the communication between multiple computing units into a high-bandwidth communication layer, a medium-bandwidth communication layer, and a low-bandwidth communication layer, the construction module 504 can construct communication topologies for the high-bandwidth communication layer, the medium-bandwidth communication layer, and the low-bandwidth communication layer, respectively, based on the hardware interconnection of the computing units within the high-bandwidth communication layer, the medium-bandwidth communication layer, and the low-bandwidth communication layer.

[0114] The apparatus 500 may also include a communication module 506. The communication module 506 may be configured to perform a communication process for each of the multiple layers based on a constructed communication topology. According to at least one embodiment of this disclosure, the communication processes at the multiple layers are executed in parallel.

[0115] For example, according to at least one embodiment of this disclosure, after constructing communication topologies for a high-bandwidth communication layer, a medium-bandwidth communication layer, and a low-bandwidth communication layer, respectively, the communication module 506 can execute communication processes at the high-bandwidth, medium-bandwidth, and low-bandwidth communication layers based on the communication topologies of the high-bandwidth, medium-bandwidth, and low-bandwidth communication layers, respectively. According to at least one embodiment of this disclosure, the communication processes at the high-bandwidth, medium-bandwidth, and low-bandwidth communication layers can be executed in parallel.

[0116] According to at least one embodiment of this disclosure, the communication bandwidth between computing units in the same level of communication topology is the same.

[0117] According to at least one embodiment of this disclosure, the construction module 504 may also be configured to construct a hierarchical communication topology based on at least one of hardware interconnection between computing units within a hierarchy, hardware interconnection between computing units within a hierarchy and network communication devices, or hardware interconnection between network communication devices within a hierarchy.

[0118] According to at least one embodiment of this disclosure, the network communication device includes at least one of a network interface card (NIC) or a switch.

[0119] According to at least one embodiment of the present disclosure, the distributed computing system includes multiple computing nodes, each computing node includes at least one computing unit, multiple layers include a first layer, the first layer corresponds to the communication layer within the computing node, and the communication topology of the first layer is used for communication between at least one computing unit within the computing node.

[0120] According to at least one embodiment of the present disclosure, the plurality of layers includes a second layer, the second layer corresponding to an inter-computing node communication layer, and the communication topology of the second layer is used for communication between the plurality of computing nodes.

[0121] According to at least one embodiment of the present disclosure, each computing node includes a plurality of sequentially numbered computing units, and in a second-level communication topology, each of the sequentially numbered computing units of each computing node directly communicates with a computing unit with the same number among the sequentially numbered computing units of another computing node.

[0122] According to at least one embodiment of this disclosure, the communication topology includes a ring communication topology or a tree communication topology.

[0123] According to at least one embodiment of the present disclosure, distributed communication is used for training or inference of a distributed model; and the distributed model includes a neural network model.

[0124] According to at least one embodiment of the present disclosure, the computing unit includes a neural network processing unit (NPU), a general-purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).

[0125] According to at least one embodiment of the present disclosure, a method, apparatus, electronic device, and storage medium for distributed communication are provided, which can simplify the structure of the communication topology, shorten the operation path, improve the utilization of communication bandwidth, and enable more flexible network configuration.

[0126] Figure 6 An electronic device according to at least one embodiment of the present disclosure is shown.

[0127] The method and apparatus for distributed communication provided in at least one embodiment of this disclosure can be applied to different systems or devices, such as those used in... Figure 6 The electronic device shown is 600.

[0128] The electronic device 600 can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, AR device, VR device, vehicle terminal, etc., or it can be a server. The method for distributed communication provided in at least one embodiment of this disclosure can be applied to scenarios involving distributed communication in the electronic device 600, such as high-performance computing (HPC) and artificial intelligence (AI), for example, a tensor computing unit. Of course, the embodiments of this disclosure are not limited to this; any scenario, device, or apparatus involving distributed communication can employ the method or apparatus for distributed communication provided in at least one embodiment of this disclosure.

[0129] In some embodiments, the apparatus for distributed communication provided in at least one embodiment of this disclosure can be a chip, such as a system-on-a-chip (SoC). The SoC includes a processor, which can be a single-core or multi-core processor, memory, and I / O interfaces, etc. The processor can load data and applications from memory and process the data, for example, for distributed communication.

[0130] It should be noted that the data involved in the method and apparatus for distributed communication provided in at least one embodiment of this disclosure may have different physical meanings depending on the application scenario. For example, the method for distributed communication provided in at least one embodiment of this disclosure can be applied in fields such as voice processing, graphics processing, text processing, and video processing.

[0131] For example, in the field of speech processing, data can be input and output parameters in tasks such as feature extraction, speech enhancement, and speech recognition.

[0132] For example, in the field of graphics processing, data can be the input and output parameters for tasks such as image recognition, feature extraction, image segmentation, object detection, image classification, and scene reconstruction.

[0133] For example, in the field of text processing, data can be the input and output parameters for tasks such as text classification, sentiment analysis, and text generation.

[0134] For example, in the field of video processing, the data can be relevant parameters from the field of graphics processing, or input and output parameters specific to the field of video processing, such as optical flow operators (used to estimate motion between video frames) and target tracking operators (used to track specific targets in video).

[0135] Of course, the embodiments disclosed herein are not limited to these. For other application scenarios or fields, as long as distributed communication is required, the method for distributed communication described in at least one embodiment of this disclosure can be applied, and will not be described in detail here.

[0136] like Figure 6 As shown, the electronic device 600 includes a processing unit 601, which can perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in a memory to achieve various functions.

[0137] For example, when the computer-readable instruction processing device 601 executes, it can perform one or more steps of the method for distributed communication according to any of the above embodiments. It should be noted that a detailed description of the process of the method for distributed communication can be found in the relevant descriptions in the embodiments of the method for distributed communication described above.

[0138] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 603 and / or cache memory, etc., for example, computer-readable instructions may be loaded from storage device 608 into RAM 603 to execute computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 602, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0139] For example, the processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0140] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, flash memory, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 6An electronic device 600 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 600 may alternatively implement or have more or fewer devices. For example, the processing device 601 may control other components in the electronic device 600 to perform desired functions.

[0141] For example, the processing device 601 can be a central processing unit (CPU), a tensor processor (TPU), or a graphics processing unit (GPU) with data processing and / or program execution capabilities. The CPU can be based on x86, ARM, RISC-V, or other architectures. The GPU can be directly integrated into the SoC, directly integrated onto the motherboard, or built into the motherboard's northbridge chip.

[0142] Figure 7 A non-transitory computer-readable storage medium 700 according to at least one embodiment of the present disclosure is shown.

[0143] For example, such as Figure 7 As shown, the storage medium 700 can be a non-transitory computer-readable storage medium on which one or more computer-readable instructions 701 can be stored non-transitory. For example, when the computer-readable instructions 701 are executed by a processor, one or more steps in the method for distributed communication described above can be performed.

[0144] For example, the storage medium 700 can be used in an electronic device 600, such as the storage medium 700 including the storage device 608 in the electronic device 600.

[0145] For example, a storage device may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and a processor may execute these instructions to perform various functions of the processor. Various application programs and various data may also be stored in the storage medium.

[0146] For example, the storage medium may include a memory card for a smartphone, a cache component for a tablet computer, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0147] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0148] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0149] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0150] Those skilled in the art will understand that the illustrative embodiments described above are not intended to be limiting. It should be understood that any two or more of the embodiments disclosed herein can be combined in any combination. Furthermore, other embodiments may be utilized and other changes may be made without departing from the spirit and scope of the subject matter presented herein. It will be readily understood that aspects of this disclosure, as generally described herein and illustrated in the accompanying drawings, can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are contemplated herein.

[0151] Various embodiments of this disclosure can be implemented as computer-readable code embodied on a computer-readable recording medium from a particular perspective. A computer-readable recording medium is any data storage device capable of storing data readable by a computer system. Examples of computer-readable recording media may include read-only memory (ROM), random access memory (RAM), optical disc read-only memory (CD-ROM), magnetic tape, floppy disk, optical data storage device, carrier wave (e.g., data transmission via the Internet), and the like. Computer-readable recording media can be distributed via computer systems connected via a network, and thus computer-readable code can be stored and executed in a distributed manner. Furthermore, the functional programs, code, and code segments used to implement the various embodiments of this disclosure can be readily interpreted by those skilled in the art applying the embodiments of this disclosure.

[0152] The medium can be read by a computer, stored in a memory, and executed by a processor. Various embodiments can be implemented by a computer or a portable terminal including a controller and a memory, and the memory can be an example of a non-transitory computer-readable recording medium suitable for storing programs(s) having instructions for implementing embodiments of this disclosure. This disclosure can be implemented by a program having code for specifically implementing the apparatus and methods described in the claims, said program being stored in a machine (or computer) readable storage medium. The program can be carried electronically on any medium, such as communication signals transmitted via wired or wireless connections, and this disclosure suitably includes its equivalents.

[0153] The embodiments described herein are not intended to be limiting. The aspects of this disclosure, as generally described herein and shown in the accompanying drawings, can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are conceivable herein. Furthermore, unless the context otherwise requires, the features shown in each drawing can be used in combination with each other. Therefore, the accompanying drawings should be considered... Figure 1 These features are generally considered to be part of one or more overall embodiments, but it should be understood that not all of the illustrated features are necessary for each embodiment.

[0154] The embodiments set forth in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Rather, they are merely some examples consistent with aspects related to the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or changes may be provided in addition to those set forth herein. For example, the above embodiments may be applicable to various combinations and sub-combinations of the disclosed features and / or combinations and sub-combinations of several other features disclosed above. Furthermore, the logical flows depicted in the drawings and / or described herein do not necessarily require the specific order or sequence shown to achieve the desired results. Other implementations are within the scope of the appended claims.

[0155] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can make various changes or substitutions within the technical scope disclosed in this disclosure, and such changes or substitutions should all be covered within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for distributed communication, characterized in that, A method applicable to a distributed computing system comprising multiple computing units, wherein the multiple computing units are communicatively connected, the method comprising: Based on the communication bandwidth between computing units, the communication between the multiple computing units is divided into multiple levels with different communication bandwidths; For each of the multiple levels: The communication topology of the layer is constructed based on the hardware interconnection of the computing units within the layer; The communication process at the aforementioned level is executed based on the constructed communication topology; The communication processes at these multiple levels are executed in parallel.

2. The method for distributed communication according to claim 1, characterized in that, The communication bandwidth between computing units in the same level of communication topology is the same.

3. The method for distributed communication according to claim 1, characterized in that, The communication topology of the layer is constructed based on the hardware interconnection of the computing units within the layer, including: The communication topology of the layer is constructed based on at least one of the following: hardware interconnection between computing units within the layer, hardware interconnection between computing units within the layer and network communication devices, or hardware interconnection between network communication devices within the layer.

4. The method for distributed communication according to claim 3, characterized in that, The network communication device includes at least one of a network interface card (NIC) or a switch.

5. The method for distributed communication according to claim 1, characterized in that, The distributed computing system includes multiple computing nodes, and each computing node includes at least one computing unit. The plurality of levels includes a first level, which corresponds to the intra-node communication level, and The first-level communication topology is used for communication between at least one computing unit within the computing node.

6. The method for distributed communication according to claim 5, characterized in that, The plurality of levels includes a second level, which corresponds to the inter-node communication level, and The second-level communication topology is used for communication between the plurality of computing nodes.

7. The method for distributed communication according to claim 6, characterized in that, Each computing node consists of multiple sequentially numbered computing units. In the second-level communication topology, each computing unit in a sequentially numbered plurality of computing units of each computing node communicates directly with the computing unit with the same number in a sequentially numbered plurality of computing units of another computing node.

8. The method for distributed communication according to claim 1, characterized in that, The communication topology includes either a ring communication topology or a tree communication topology.

9. The method for distributed communication according to claim 1, characterized in that, The distributed communication is used for training or inference of the distributed model; and The distributed model includes a neural network model.

10. An apparatus for distributed communication, characterized in that, The device includes: The module is configured to divide multiple computing units into multiple levels with different communication bandwidths based on the communication bandwidth between computing units in a distributed computing system. The building module is configured to construct the communication topology of each of the multiple layers based on the hardware interconnection of the computing units within that layer. The communication module is configured to perform the communication process of each of the plurality of layers based on the constructed communication topology; The communication processes at these multiple levels are executed in parallel.

11. An electronic device for distributed communication, characterized in that, The electronic device includes: One or more processors; Memory, which stores one or more computer programs; Wherein, when executed by the one or more processors, the one or more computer programs cause the one or more processors to implement the method as described in any one of claims 1-9.

12. A non-transitory computer-readable storage medium, characterized in that, It stores computer-executable instructions thereon, which, when executed by a processor, cause at least one processor to implement the method as described in any one of claims 1-9.

Images