Collective communication method and apparatus

Abstract

Provided are a collective communication method and apparatus. The method is applied to a distributed computing system, and comprises: acquiring a task to be computed, wherein the task to be computed comprises a first-type parallel computing task and a second-type parallel computing task that have no dependency relationship; with respect to the first-type parallel computing task and the second-type parallel computing task, respectively creating a first-type communication domain and a second-type communication domain, wherein the first-type communication domain and the second-type communication domain at least comprise two identical computing devices, and each of the first-type communication domain and the second-type communication domain has an independent communication buffer and an independent communication connection; and concurrently executing a collective communication operation for the first-type communication domain and a collective communication operation for the second-type communication domain. The collective communication method provided in the present application can create an independent communication buffer and an independent communication connection for each of a plurality of communication domains that can be concurrently executed, thereby realizing concurrent execution of the plurality of communication domains, avoiding communication connection preemption, and improving the execution efficiency of a distributed computing task.

Description

A collection communication method and device

[0001] The present application claims priority from the Chinese patent application No. 202410852119.5, filed on June 27, 2024, and entitled "A collection communication method and device", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of information technology, and in particular to a collection communication method and device. BACKGROUND

[0003] In the artificial intelligence (AI) large model training business, due to the model parameter quantity being restricted by the single device memory space and the massive training data being restricted by the single device computing power, distributed computing power devices need to be used for parallel processing. Commonly used distributed parallel methods mainly include data parallelism (DP), tensor parallelism (TP), pipeline parallelism (PP), expert parallelism (EP), and sequence parallelism (SP). In the large model training process, multiple parallel methods are often mixed and used, and the computing power devices under each parallel method need to use a collection communication method for communication and cooperation.

[0004] In the time dimension, the collection communication under data parallelism (DP) has no sequential dependency relationship with the collection communication under other parallel methods. The backward calculation process of a general neural network (which will be interleaved with PP, EP, SP, and TP communication) and the gradient aggregation process DP are executed concurrently. When the collection communication in the DP domain and the collection communication in the PP, EP, SP, and TP domains use the same communication connection, there is a communication connection preemption problem. SUMMARY

[0005] Embodiments of the present application provide a collection communication method and device. In a plurality of communication domains that can be executed concurrently, an independent communication buffer and a communication connection are created for each communication domain, the concurrent execution of the plurality of communication domains is realized, the communication connection preemption problem is avoided, and the execution efficiency of a distributed computing task is improved.

[0006] In a first aspect, the application provides a collective communication method, which is applied to a distributed computing system including N computing devices, N being a positive integer greater than 1. The collective communication method includes obtaining a to-be-computed task, the to-be-computed task including a first type of parallel computing task and a second type of parallel computing task, and the first type of parallel computing task and the second type of parallel computing task having no dependency relationship; creating a first type of communication domain and a second type of communication domain for the first type of parallel computing task and the second type of parallel computing task, respectively, wherein the first type of communication domain and the second type of communication domain include at least two same computing devices, each communication domain in the first type of communication domain and the second type of communication domain has an independent communication buffer and a communication connection, the first type of communication domain is used to perform a collective communication operation of a service data stream of the first type of parallel computing task, and the second type of communication domain is used to perform a collective communication operation of a service data stream of the second type of parallel computing task; and concurrently performing the collective communication operation of the first type of communication domain and the collective communication operation of the second type of communication domain.

[0007] The collective communication method provided by the application creates multiple communication domains for multiple parallel computing tasks that can be concurrently executed or multiple parallel computing tasks that can be potentially concurrently executed, and the multiple communication domains are each configured with an independent communication buffer and a communication connection, so that the collective communication operations of the multiple communication domains can be concurrently executed, the problem of communication connection preemption is overcome, and the execution efficiency of the distributed computing task is improved.

[0008] In one possible implementation, the collective communication method provided by the application can be applied to a distributed training task of an AI model, that is, the to-be-computed task is a distributed training task, the first type of parallel computing task includes a DP computing task, and the second type of parallel computing task includes at least one of a PP computing task, a TP computing task, an SP computing task, and an EP computing task. In this case, the first type of communication domain can be referred to as a DP communication domain, and the second type of communication domain can be referred to as a PP communication domain, a TP communication domain, an SP communication domain, or an EP communication domain.

[0009] For example, the reverse computation process of the AI model and the gradient aggregation process have no computational dependency and can be concurrently executed. The reverse computation process will interleave the execution of the PP computing task, the EP computing task, the SP computing task, and the TP computing task, and the gradient aggregation is implemented by the DP computing task, so the DP computing task and the PP computing task, the EP computing task, the SP computing task, and the TP computing task can be concurrently executed.

[0010] The application establishes independent communication resources (including communication buffers and communication connections) for the DP communication domain and the PP communication domain, the TP communication domain, the SP communication domain, and the EP communication domain respectively, realizes concurrent execution of the set communication operation of the DP communication domain and the set communication operation of the PP communication domain, the TP communication domain, the SP communication domain, and the EP communication domain, avoids the problem of communication connection preemption, and improves the distributed training task execution efficiency of the AI model.

[0011] In another possible implementation, in the distributed training task scenario of the AI model, the N computing devices can be N hardware accelerators, and the N hardware accelerators are distributed in one server or distributed in multiple servers.

[0012] In another possible implementation, each communication domain is determined based on the splitting manner of the M computing devices corresponding to each parallel computing task, the M computing devices are the computing devices available to the AI model in the distributed training task of the AI model, and M is a positive integer less than or equal to N.

[0013] For example, the number of hardware accelerators allocated by the distributed computing system to the distributed training task of the AI model is 32, the distributed training task of the AI model uses DP computing, PP computing, and TP computing, the splitting manner of the 32 hardware accelerators for the DP computing is every 2 as a group, which is split into 16 parts to form 16 DP groups, and every 2 hardware accelerators constitute a DP communication domain, the splitting manner of the 32 hardware accelerators for the PP computing is every 4 as a group, which is split into 8 parts to form 8 PP groups, and every 4 hardware accelerators constitute a PP communication domain, and the splitting manner of the 32 hardware accelerators for the TP computing is every 4 as a group, which is split into 8 parts to form 8 TP groups, and every 4 hardware accelerators constitute a TP communication domain.

[0014] Therefore, it can be known that in the distributed training process of the AI model, one hardware accelerator belongs to the DP communication domain, the PP communication domain, and the TP communication domain at the same time, but one hardware accelerator only has one communication buffer (also called a transceiver buffer) and one communication connection with other hardware accelerators, and when the DP communication domain and the PP communication domain and the TP communication domain communicate concurrently, the communication buffer and the communication connection will be preempted. The set communication method provided by the application respectively creates independent communication buffers and communication connections for the DP communication domain and the PP communication domain and the TP communication domain, avoids the problem that the communication connection is preempted when the DP communication domain and the PP communication domain and the TP communication domain communicate concurrently, and improves the distributed training task execution efficiency of the AI model.

[0015] The communication buffer of each communication domain can be a shared memory, a remote direct memory access memory region (RDMA MR) memory, and a memory buffer of a socket protocol stack, and the communication connection of each communication domain can be a RDMA queue pair (QP) and a socket.

[0016] It can be understood that the type of communication buffer and the communication connection mode of each communication domain depend on the actual application scenario. For example, for a scenario where the communication domain does not cross servers, i.e., the hardware accelerators corresponding to the communication domain are all in the same server, the communication domain uses a shared memory as the communication buffer, and the communication mode between different hardware accelerators in the communication domain is connectionless, and each hardware accelerator realizes communication by accessing its own shared memory. For a scenario where the communication domain crosses servers, i.e., the hardware accelerators corresponding to the communication domain are partially in one server and partially in another server, the communication domain can use a RDMA MR memory as the communication buffer, and the communication connection mode of the hardware accelerators that cross servers in the communication domain is a RDMA QP, or the communication domain can use a memory buffer of a socket protocol stack as the communication buffer, and the communication mode of the hardware accelerators that cross servers in the communication domain is a socket.

[0017] The size of the memory buffer of each communication domain is determined based on the size of the service data stream of the parallel computing task corresponding to each communication domain. For example, for a hardware accelerator that is shared by multiple communication domains, such as hardware accelerator A that is shared by DP communication domain, PP communication domain, TP communication domain, and EP communication domain, since the DP service data stream corresponding to the DP communication domain is the largest, the communication buffer size configured for the DP communication domain is the largest, the TP service data stream corresponding to the TP communication domain and the EP service data stream corresponding to the EP communication domain are next, and then the communication buffer size configured for the TP communication domain and the EP communication domain is next, and the PP service data stream corresponding to the PP communication domain is the smallest, and then the communication buffer size configured for the PP communication domain is the smallest. In this way, the size of the communication buffer of each communication domain is optimized based on the size characteristics of the service data stream processed by each communication domain.

[0018] In another possible implementation, the communication connection of each communication domain is configured with a priority parameter, and the priority parameter indicates the scheduling priority of the set operation of the service data stream of each communication domain when the set operation of each communication domain is concurrently executed.

[0019] By configuring the priority of the communication connection of each communication domain, the communication domain level priority configuration capability is realized. When the set communication operations of multiple communication domains are concurrently executed, the set communication operations of multiple communication domains are scheduled according to the priority, more set communication operations of important service data streams are scheduled for execution, and less set communication operations of unimportant service data streams are scheduled, thereby optimizing the execution of the overall computing task.

[0020] In another possible implementation, the priority parameter of the communication connection of each communication domain is determined based on the dependency relationship of the set communication operation of each communication domain and the operator of the reverse calculation in the distributed training task of the AI model.

[0021] For example, the execution of the reverse calculation task (also referred to as an operator) has a strong dependency on the set communication operation of the EP communication domain, the execution of the reverse calculation task has a strong dependency on the set communication operation of the TP communication domain, and the execution of the reverse calculation task has a strong dependency on the set communication operation of the SP communication domain. The priority parameter of the communication connection of the EP communication domain, the TP communication domain, and the SP communication domain is configured as high (high). The execution of the reverse calculation task has a weak dependency on the communication operation of the PP communication domain, and the priority parameter of the communication connection of the PP communication domain is configured as middle (middle). The execution of the DP calculation task has no dependency on the set communication operation of the DP communication domain, and the priority parameter of the communication connection of the DP communication domain is configured as low (low).

[0022] In another possible implementation, a specific implementation of concurrently executing the set communication operation of the first type of communication domain and the set communication operation of the second type of communication domain is as follows: based on the service data stream of the set communication operation to be executed by each communication domain, a plurality of data transmission requests corresponding to each communication domain are generated; based on the priority parameter of the communication connection of each communication domain, the scheduling proportion of the plurality of data transmission requests corresponding to each communication domain is determined; based on the scheduling proportion, the target data transmission request corresponding to each communication domain is scheduled; and the target data transmission request corresponding to the first type of communication domain and the target data transmission request corresponding to the second type of communication domain are concurrently executed.

[0023] By providing the communication domain level communication scheduling priority configuration capability for multiple communication domains, when the set communication operations of multiple communication domains are concurrently executed, the service data streams of each communication domain are scheduled according to the priority of each communication domain, the service data streams of the communication domain with high priority are scheduled more, and the service data streams of the communication domain with low priority are scheduled less, thereby optimizing the concurrent execution of each communication domain, filling the bandwidth of the physical communication link with the concurrent communication of the service data streams of multiple communication domains, and improving the bandwidth utilization rate.

[0024] In a second aspect, the present application provides a collective communication device deployed in a distributed computing system, the distributed computing system comprising N computing devices, N being a positive integer greater than 1, the collective communication device comprising an obtaining module, a creating module and an executing module, wherein the obtaining module is configured to obtain a to-be-computed task, the to-be-computed task comprising a first type of parallel computing task and a second type of parallel computing task, the first type of parallel computing task and the second type of parallel computing task having no dependency relationship; the creating module is configured to create a first type of communication domain and a second type of communication domain for the first type of parallel computing task and the second type of parallel computing task respectively, wherein the first type of communication domain and the second type of communication domain comprise at least two same computing devices, each communication domain in the first type of communication domain and the second type of communication domain has an independent communication buffer and a communication connection, the first type of communication domain is configured to perform a collective communication operation of a service data stream of the first type of parallel computing task, and the second type of communication domain is configured to perform a collective communication operation of a service data stream of the second type of parallel computing task; and the executing module is configured to concurrently execute the collective communication operation of the first type of communication domain and the collective communication operation of the second type of communication domain.

[0025] In a possible implementation, the to-be-computed task is a distributed training task of an AI model; the first type of parallel computing task comprises a data parallel computing task, and the second type of parallel computing task comprises at least one of a pipeline parallel computing task, a tensor parallel computing task, a sequence parallel computing task and an expert parallel computing task.

[0026] In another possible implementation, the N computing devices comprise N hardware accelerators, and the N hardware accelerators are distributed in one server or distributed in multiple servers.

[0027] In another possible implementation, each communication domain is determined based on a splitting manner of M computing devices by a parallel computing task corresponding to each communication domain, the M computing devices being computing devices available for the distributed training task of the AI model in the N computing devices, and M being a positive integer less than or equal to N.

[0028] In another possible implementation, a size of a memory buffer of each communication domain is determined based on a size of a service data stream of a parallel computing task corresponding to each communication domain.

[0029] In another possible implementation, a communication connection of each communication domain is configured with a priority parameter, the priority parameter indicating a scheduling priority of a collective communication operation of a service data stream of each communication domain when the collective communication operations of each communication domain are concurrently executed.

[0030] In another possible implementation, the priority parameter of the communication connection of each communication domain is determined based on a dependency relationship between an operator of a parallel computing task corresponding to each communication domain and a collective communication operation of each communication domain.

[0031] In another possible implementation, the execution module is specifically used to: generate multiple data transmission requests corresponding to each communication domain based on the business data stream of the set communication operations to be performed in each communication domain; determine the scheduling ratio of the multiple data transmission requests corresponding to each communication domain based on the priority parameters of the communication connections of each communication domain; schedule the target data transmission requests corresponding to each communication domain based on the scheduling ratio; and concurrently execute the scheduled target data transmission requests corresponding to the first type of communication domain and the target data transmission requests corresponding to the second type of communication domain.

[0032] Thirdly, embodiments of this application provide a computing device, including a memory and a processor, wherein the memory stores instructions that, when executed by the processor, cause the method described in the first aspect to be implemented.

[0033] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the method described in the first aspect to be implemented.

[0034] Fifthly, embodiments of this application also provide a computer program or computer program product, the computer program or computer program product including instructions that, when executed, cause a computer to perform the method described in the first aspect.

[0035] In a sixth aspect, embodiments of this application also provide a chip including at least one processor and a communication interface, wherein the processor is used to execute the method described in the first aspect. Attached Figure Description

[0036] Figure 1 illustrates the parallelism used in the distributed training of an AI model, and a schematic diagram of how the various parallelisms are divided in the hardware accelerator cluster.

[0037] Figure 2 illustrates the communication link preemption problem that occurs when DP ensemble communication and other parallel communication methods are executed in parallel during the distributed training of an AI model.

[0038] Figure 3 illustrates a schematic diagram of the queued serial execution of DP set communication and other parallel communication methods during the distributed training process of AI models in related technologies.

[0039] Figure 4 shows a schematic diagram of a distributed training architecture for an AI model;

[0040] Figure 5 is a flowchart illustrating a collection communication method provided in an embodiment of this application;

[0041] Figure 6 shows a schematic diagram of a communication domain created by a collection communication library according to an embodiment of this application;

[0042] FIG. 7 shows a schematic diagram of communication buffer sizes of different communication domains created by a collective communication library according to an embodiment of the present application;

[0043] FIG. 8 shows a signaling interaction diagram of distributed training of an AI model using the collective communication method according to an embodiment of the present application;

[0044] FIG. 9 shows a schematic diagram of parallel execution of DP collective communication and communication in other parallel modes after using the collective communication method according to an embodiment of the present application;

[0045] FIG. 10 shows a structural schematic diagram of a collective communication apparatus according to an embodiment of the present application;

[0046] FIG. 11 shows a structural schematic diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] The term “and / or” mentioned herein is a description of an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The symbol “ / ” herein represents an or relationship of associated objects, for example, A / B represents A or B.

[0048] The terms “first” and “second” and the like in the description and claims herein are used to distinguish different objects, and are not used to describe a specific order of the objects. For example, the first type of parallel computing task and the second type of parallel computing task are used to distinguish different types of parallel computing tasks, and are not used to describe a specific order of the parallel computing tasks.

[0049] In the embodiments of the present application, the words “exemplary” or “for example” are used to mean serving as an example, instance, or illustration. Any embodiment or design scheme described as “exemplary” or “for example” in the embodiments of the present application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Rather, the use of “exemplary” or “for example” is intended to present relevant concepts in a concrete manner.

[0050] In the description of the embodiments of the present application, unless otherwise specified, “a plurality of” means two or more, for example, a plurality of computing devices means two or more computing devices, and the like; a plurality of hardware accelerators means two or more hardware accelerators, and the like.

[0051] The collection communication method and device provided in the embodiments of the present application can be applied to a complex parallel strategy combination scenario in a distributed computing system, multiple parallel computing tasks have no computational dependency relationship and can be logically executed concurrently, a communication domain is created for each parallel computing task, a communication buffer and a communication connection at the communication domain level are provided, concurrent execution of collection communication operations of multiple communication domains is supported, the communication connection preemption problem between the collection communication operations of multiple communication domains is overcome, and the computing efficiency of distributed computing is improved.

[0052] For example, the collection communication method and device provided in the embodiments of the present application can be applied to the field of high performance computers (HPC). HPC is a technology that uses a powerful processor cluster to process massive multi-dimensional data sets in parallel and solves complex problems at extremely high speed. For example, HPC can be used to solve relatively complex problems such as DNA sequencing, automated trading and fraud detection in finance, weather forecasting, seismic data processing, geospatial analysis, wind simulation, etc. The HPC supercomputer usually uses a complex parallel strategy combination to process complex problem computing tasks, that is, multiple parallel computing tasks. The collection communication method provided in the present application can be applied to parallel computing tasks that have no dependency relationship in the multiple parallel computing tasks, communication resources at the communication domain level are provided, parallel execution is performed, the communication connection preemption problem is avoided, and the processing efficiency of the overall computing task is improved.

[0053] For another example, the collection communication method and device provided in the embodiments of the present application can also be applied to distributed training tasks of an AI model. The collection communication method provided in the present application creates independent communication resources for each communication domain corresponding to a parallel degree, avoids the communication connection preemption problem when multiple communication domains perform collection communication operations in parallel, and improves the training efficiency of the AI model.

[0054] The following describes the collection communication method and device provided in the embodiments of the present application by taking the distributed training scenario of an AI model as an example. The implementation of the HPC supercomputer scenario is similar, and the specific implementation can be referred to the implementation of the collection communication method and device of the distributed training scenario of the AI model.

[0055] In actual application, when the parameter quantity in the AI model to be trained and the sample data quantity for training the AI model are large, the limited computing power of a single device can be difficult to complete the training of the AI model alone. Therefore, the computing power of multiple devices can be integrated to jointly train the AI model in a distributed training manner. The device for training the AI model can be a processor-level device, such as a neural network processing unit (NPU), a graphics processing unit (GPU), or the like. Alternatively, the device for training the AI model can be a chip-level device, such as a plurality of chips connected to a host. Alternatively, the device for training the AI model can be a server-level device, such as a plurality of independent servers. When the multiple devices for training the AI model are processor-level devices or chip-level devices, the multiple processors can be located in the same server (which can constitute a computing node) or in different servers. When the multiple devices for training the AI model are server-level devices, the multiple devices can be located in the same data center (which can be regarded as a computing node), or the multiple devices can be located in different data centers, i.e., the AI model can be distributed trained across data centers.

[0056] Embodiments of the present application take a computing device as an example of a processor level to introduce distributed training of an AI model. The distributed training system includes a plurality of hardware accelerators (e.g., a plurality of NPUs). The plurality of hardware accelerators can be distributed in the same server (which can be referred to as single-machine multi-card at this time) or in multiple servers (which can be referred to as multi-machine multi-card at this time).

[0057] FIG. 1 shows the parallelism adopted in the distributed training of an AI model and the splitting manner of each parallelism for a hardware accelerator cluster. The blocks in FIG. 1 represent hardware accelerators, and the blocks with different labels represent hardware accelerators of different TP groups. The hardware accelerator cluster performing the distributed training task of the AI model includes 32 hardware accelerators. For the hardware accelerator cluster composed of 32 hardware accelerators, the DP splitting manner is to divide into 16 groups of 2, split into 16 parts, form 16 DP groups, and each 2 hardware accelerators constitute a DP communication domain; the PP splitting manner is to divide into 8 groups of 4, split into 8 parts, form 8 PP groups, and each 4 hardware accelerators constitute a PP communication domain; and the TP calculation splitting manner for 32 hardware accelerators is to divide into 8 groups of 4, split into 8 parts, form 8 TP groups, and each 4 hardware accelerators constitute a TP communication domain.

[0058] Figure 2 shows a schematic diagram of the communication connection preemption problem when the DP set communication and the communication in other parallel modes are executed in parallel in the AI model distributed training process. As shown in Figure 2, in the time dimension, the set communication in the DP and the set communication in other parallel modes have no sequential dependency. The back propagation process of the general neural network (which will be interleaved with PP, EP, SP, and TP communication) and the gradient aggregation process DP are executed concurrently. When the set communication in the DP communication domain and the set communication in the PP, EP, SP, and TP communication domain use the same communication connection, there is a communication connection preemption problem.

[0059] As shown in Figure 1, the same hardware accelerator belongs to the communication domains of various parallel degrees used in the distributed training of the AI model, such as the distributed parallel modes used in the distributed training of the AI model, including DP, PP, and TP. According to the splitting of the hardware accelerator cluster by DP, PP, and TP, DP, PP, and TP communication domains are created, and the same hardware accelerator belongs to the DP, PP, and TP communication domains. For example, the DP communication domain includes hardware accelerator a and hardware accelerator b, the PP communication domain includes hardware accelerator a, hardware accelerator b, hardware accelerator c, and hardware accelerator d, and the TP communication domain includes hardware accelerator a, hardware accelerator b, hardware accelerator c, and hardware accelerator d. At this time, the DP, PP, and TP communication domains all include hardware accelerator a and hardware accelerator b. When the set operation of the DP communication domain, the communication operation of the PP communication domain, and the communication operation of the TP communication domain are executed concurrently, it will inevitably cause the preemption of the communication resources (including the communication buffer and the communication connection of hardware accelerator a) of hardware accelerator a and hardware accelerator b.

[0060] Figure 3 shows a schematic diagram of the queuing and serial execution when the DP set communication and the communication in other parallel modes occur simultaneously in the AI model distributed training process in the related art.

[0061] As shown in Figure 3, the training scheme in the related art is to concurrently execute the TP, EP, PP, SP, and other service streams in the back propagation process and the DP gradient summation service stream in the time dimension. When the set communication operation of the DP communication domain and the set communication operation of the TP, EP, PP, and SP communication domain occur simultaneously, the set communication operation of the DP communication domain and the set communication operation of the TP, EP, PP, and SP communication domain need to be queued and executed in series.

[0062] Currently, the mainstream set communication library (such as NCCL, OpenMPI, etc.) uses a shared communication connection resource mode to implement the communication between various communication domains (such as the communication domain where DP is located and the communication domain where PP is located). The set communication between different communication domains needs to be sequentially issued and executed on different hardware accelerators (such as NPUs), and the potentially concurrent set communication operations in multiple communication domains need to be queued and executed in series.

[0063] Although the communication connection preemption problem of the set communication operation of different communication domains is avoided in the related art, the following problems exist:

[0064] 1. A large amount of data flow of low priority (such as a DP communication service flow) blocks a small amount of data flow of higher priority (such as a TP|EP|SP|PP communication service flow).

[0065] 2. The small amount of data flow blocks the execution of the large amount of data flow, but is affected by factors such as scheduling synchronization, resulting in insufficient utilization of the bandwidth of the communication link. For example, the communication data volume in the SP dimension is small, and at this time the communication link has extra bandwidth that can be given to the DP dimension communication service flow for execution.

[0066] 3. The serial mechanism of the set communication operation of different communication domains can cause idle time of computing resources in the time dimension and insufficient utilization.

[0067] To this end, the embodiment of the present application provides a set communication method, which re-plans the scheduling execution strategy of each communication domain on different parallel degrees, creates an independent memory buffer and a communication connection for each communication domain, and implements concurrent scheduling execution of the set communication operation between multiple communication domains. Meanwhile, the embodiment of the present application provides a communication domain level memory buffer and a service flow priority configuration capability according to the traffic characteristics of different service flows in the large model training business, and realizes the concurrent optimization of reverse calculation and each parallel dimension communication in a complex parallel strategy combination scenario.

[0068] The specific implementation of the set communication method and device provided by the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0069] FIG. 4 shows a schematic diagram of a distributed training architecture of an AI model. As shown in FIG. 4, the distributed training architecture of an AI model includes, from bottom to top, a distributed AI training machine hardware layer, a device driver, a computing architecture layer (including a runtime library, a graph engine, an operator development kit, an operator library, a set communication library, and a tool chain), a deep learning framework layer, and a large model distributed training business layer.

[0070] The distributed AI training machine hardware layer includes a plurality of hardware accelerators, such as a plurality of NPUs or a plurality of GPUs, and other devices such as a memory, a network card, etc., for providing data processing capability for the upper layer and supporting the services of the upper layer.

[0071] The device driver layer is used to respond to the call of the computing architecture to the hardware layer, and uses a plurality of processors in the hardware layer to perform corresponding data processing operations, such as parallelizing the training of an AI model using a plurality of processors in the hardware layer.

[0072] The computing architecture layer is configured to provide a programming interface for development and deployment, support users to quickly build AI applications and businesses based on AI models, and invoke the parallelization capability of the plurality of processors in the hardware layer to implement AI model training. Further, the computing architecture can also implement functions such as graph-level and operator-level compilation optimization and automatic tuning of the AI model.

[0073] The computing architecture layer includes a runtime library, a graph engine, an operator development kit, an operator library, a collective communication library, and a tool chain. The runtime library is configured to provide functions such as resource management, media data preprocessing, and model inference, and support developers to quickly build AI applications. The graph engine is configured to be a computing graph compilation and running control center, and provide graph compilation optimization and loading execution capability. The operator development kit is configured to be for operator development scenarios, match user development habits, and support operator programming. The operator library is configured to provide a rich depth optimization and hardware affinity high-performance operator. The collective communication library is configured to provide a single-machine multi-card and multi-machine multi-card data parallel, model parallel collective communication scheme. The tool chain is a collection of various programming tools, which form a tool chain for developing application programs and operating systems.

[0074] The deep learning framework layer is configured to integrate development components and pre-trained models, shield user perception of underlying complex hardware, and provide a service for users to quickly develop AI models. Exemplarily, the deep learning framework can be a TensorFlow framework, a PyTorch framework, or a MindSpore framework, or can be another type of deep learning framework, which is not limited in the embodiments of the present application.

[0075] The collective communication method provided in the embodiments of the present application is mainly an improvement of the collective communication library in FIG. 4. In the distributed training of the AI model, a communication connection at the communication domain level is created for each parallel degree, and the collective communication of each parallel degree can be executed in parallel, thereby overcoming the problem of communication link preemption and improving the efficiency of the distributed training of the AI model.

[0076] FIG. 5 is a flowchart of a collective communication method according to an embodiment of the present application. As shown in FIG. 5, the collective communication method provided in the embodiments of the present application includes at least steps S501 to S503.

[0077] In step S501, an AI model training task to be processed is obtained.

[0078] In an actual application scenario, when a business user develops an AI application on a deep learning framework, the user can provide an AI model for implementing the AI application to the deep learning framework, and the deep learning framework generates a distributed training task of the AI model and obtains the AI model training task.

[0079] For example, a business user can write a training script on a deep learning framework, which can be integrated with a file of an AI model built by the user on the deep learning framework. Then, the deep learning framework can parse the AI model according to the training script and perform distributed training on the AI model according to the model training logic indicated by the training script.

[0080] The distributed training task of the AI model includes multiple parallel computing tasks, such as DP parallel, TP parallel, PP parallel, SP parallel, and EP parallel. In the training process of the AI model, the backward calculation process of the AI model and the gradient aggregation process have no computational dependency and can be executed concurrently. The backward calculation process of the AI model will perform PP calculation, EP calculation, SP calculation, and TP calculation in an interleaved manner, and the gradient aggregation process will perform DP calculation. Therefore, there is no dependency between DP calculation (i.e., the first type of parallel calculation) and PP\EP\SP\TP calculation (i.e., the second type of parallel calculation), and they can be executed concurrently.

[0081] That is, there can be one or more second type of parallel calculations, which are determined according to the actual application scenario. For example, if the parallel degrees applied to the distributed training of the AI model only include DP parallel and TP parallel, the second type of parallel calculation only includes TP calculation. If the parallel degrees applied to the distributed training of the AI model only include DP parallel, TP parallel, and SP parallel, the second type of parallel calculation includes TP calculation and SP calculation.

[0082] In step S502, a first type of communication domain and a second type of communication domain are created for the first type of parallel computing task and the second type of parallel computing task, respectively. The first type of communication domain and the second type of communication domain include at least two same computing devices, and each communication domain in the first type of communication domain and the second type of communication domain has an independent communication buffer and a communication connection.

[0083] The execution of the DP computing task requires DP set communication, and the execution of the PP\EP\SP\TP computing task requires PP\EP\SP\TP set communication. Therefore, in order to ensure the implementation of DP set communication and PP\EP\SP\TP set communication, a DP communication domain is created for the DP computing task, and a PP\EP\SP\TP communication domain is created for the PP\EP\SP\TP computing task.

[0084] The communication domain of each parallel computing can be determined according to the topology of the hardware accelerator cluster and the splitting mode of each parallel computing applied to the distributed training task of the AI model.

[0085] Optionally, the splitting manner of each parallel degree can be determined by a user. For example, the deep learning framework provides an interactive interface for the user, the interactive interface provides the hardware accelerators possessed by the hardware accelerator cluster, for example, 32 hardware accelerators, and the user inputs the splitting manner of each parallel degree to the 32 hardware accelerators by interacting with the interactive interface. For example, the distributed training of the AI model includes DP computation, PP computation and TP computation, and the splitting manner input by the user through the interactive interface is that the splitting manner for the DP computation is that each 2 hardware accelerators form a group, and the 16 hardware accelerators are split into 16 DP groups; the splitting manner for the PP computation is that each 4 hardware accelerators form a group, and the 32 hardware accelerators are split into 8 PP groups; and the splitting manner for the TP computation is that each 4 hardware accelerators form a group, and the 32 hardware accelerators are split into 8 TP groups.

[0086] The deep learning framework applies to the collective communication library to create the DP communication domain, the PP communication domain and the TP communication domain for the DP computation, the PP computation and the TP computation. The collective communication library creates the DP communication domain, the PP communication domain and the TP communication domain for the DP computation, the PP computation and the TP computation respectively according to the splitting manner of the DP communication domain, the PP communication domain and the TP communication domain. For example, the splitting manner of the DP computation is that each 2 hardware accelerators form a group, and the 16 hardware accelerators are split into 16 DP groups, so each 2 hardware accelerators form a DP communication domain; the splitting manner of the PP computation is that each 4 hardware accelerators form a group, and the 32 hardware accelerators are split into 8 PP groups, so each 4 hardware accelerators form a PP communication domain; and the splitting manner of the TP computation is that each 4 hardware accelerators form a group, and the 32 hardware accelerators are split into 8 TP groups, so each 4 hardware accelerators form a PP communication domain.

[0087] As shown in FIG. 1, each hardware accelerator belongs to different communication domains according to the splitting manner of each parallel degree. However, each hardware accelerator usually includes only one communication buffer and has only one communication connection with other hardware accelerators. When the DP collective communication and the TP / PP collective communication are concurrently executed in the reverse computation of the AI model, the communication resource will be preempted, that is, when the DP collective communication operation and the TP collective communication operation are concurrently executed, the DP collective communication operation and the TP collective communication operation will preempt the communication connection between the hardware accelerator and other hardware accelerators. For example, the TP collective operation preempts the communication connection, and the DP collective communication operation must wait until the TP collective communication operation is completed, and then the DP collective communication operation is executed by using the communication connection.

[0088] The collective communication library of the embodiment of the present application supports the creation of communication resources at the communication domain level, that is, independent communication resources are created for each communication domain, including the DP communication domain, the SP communication domain, the EP communication domain, the TP communication domain and the PP communication domain.

[0089] FIG. 6 shows a schematic diagram of a communication domain created by a collection communication library according to an embodiment of the present application. As shown in FIG. 6, the hardware accelerator is NPU, and taking NPU 0 and NPU 1 as an example, a communication buffer is created for each communication domain in each NPU, and a communication connection is established for each communication domain, that is, an independent communication buffer and communication connection are created for each communication domain. For example, the communication domains include communication domain A, communication domain B, and communication domain C, and each of the communication domain A, the communication domain B, and the communication domain C includes NPU 0 and NPU 1. For the communication domain A, a communication buffer A is created for the communication domain A on NPU 0 and NPU 1 respectively, and an independent communication connection A is created for the communication domain A, that is, a communication connection A between the communication buffer A on NPU 0 and the communication buffer A on NPU 1 is established. For the communication domain B, a communication buffer B is created for the communication domain B on NPU 0 and NPU 1 respectively, and an independent communication connection B is created for the communication domain B, that is, a communication connection B between the communication buffer B on NPU 0 and the communication buffer B on NPU 1 is established. For the communication domain C, a communication buffer C is created for the communication domain C on NPU 0 and NPU 1 respectively, and an independent communication connection C is created for the communication domain C, that is, a communication connection C between the communication buffer C on NPU 0 and the communication buffer C on NPU 1 is established. In this way, when the collection communication operations of multiple communication domains are concurrently executed, the communication connection resources of the respective communication domains are used for communication to implement the respective collection communication operations. For example, when the collection communication operation of the communication domain A and the collection communication operation of the communication domain B are concurrently executed, the collection communication operation of the communication domain A is implemented by using the communication buffer A and the communication connection A (for example, socket A) of the communication domain A, and the collection communication operation of the communication domain B is implemented by using the communication buffer B and the communication connection B (for example, socket B) of the communication domain B, so that the problem of communication resource preemption when the collection communication operation of the communication domain A and the collection communication operation of the communication domain B are concurrently executed is avoided.

[0090] It can be understood that the type of buffer and the communication connection mode of the communication buffer of each communication domain depend on the actual application scenario. For example, for a scenario in which the communication domain does not cross servers, that is, a scenario in which NPU 0 and NPU 1 are in the same server, the communication domain uses shared memory as the communication buffer, the communication mode between NPU 0 and NPU 1 is connectionless, and NPU 0 and NPU 1 communicate by accessing the shared memory. For a scenario in which the communication domain crosses servers, that is, a scenario in which NPU 0 is in one server and NPU 1 is in another server, the communication domain can use RDMA MR memory as the communication buffer, and the communication connection mode of NPU 0 and NPU 1 is RDMA QP, or the communication domain can use a socket protocol stack memory buffer as the communication buffer, and the communication mode of NPU 0 and NPU 1 is socket.

[0091] In another example, different sizes of communication buffers can be created for different parallel degrees of communication domains, since the sizes of the service data flows of different parallel degrees are different. The sizes of the memory buffer areas of the DP communication domain, and the TP communication domain, the PP communication domain, the EP communication domain, and the SP communication domain are determined based on the sizes of the service data flows of the parallel computing tasks corresponding to the respective communication domains.

[0092] FIG. 7 shows a schematic diagram of the sizes of the communication buffers of the different communication domains created by the collection communication library according to an embodiment of the present application. As shown in FIG. 6, the distributed training of the AI model applies DP computing, PP computing, TP computing, and EP computing, and the buffer area of the DP communication domain created for the DP computing, the PP communication domain created for the PP computing, the TP communication domain created for the TP computing, and the EP communication domain created for the EP computing are the largest, the smallest, and the medium, respectively.

[0093] Since the service data flow in the DP communication domain is the largest, the DP communication buffer created by the NPU 0 and the NPU 1 for the DP communication domain is the largest, the service data flow in the PP communication domain is the smallest, the PP communication buffer created by the NPU 0 and the NPU 1 for the PP communication domain is the largest, and the service data flows in the TP communication domain and the EP communication domain are medium, the TP communication buffer and the EP communication buffer created by the NPU 0 and the NPU 1 for the TP communication domain and the EP communication domain are medium.

[0094] For example, the distributed training of the AI model applies DP computing, PP computing, TP computing, and EP computing, and the DP communication domain, the PP communication domain, the TP communication domain, and the EP communication domain are created for the DP computing, the PP computing, the TP computing, and the EP computing, respectively. Taking the hardware accelerator as the NPU 0 and the NPU 1 as an example, the memory size of the NPU 0 and the NPU 1 is 64 GB, and the communication buffers are created for the respective communication domains according to the sizes of the service data flows of the respective communication domains. Since the service data flow in the DP communication domain is the largest, the communication buffer created for the DP communication domain is 300 MB to ensure the communication performance of the DP communication domain; the service data flows in the TP communication domain and the EP communication domain are medium, and the communication buffers created for the TP communication domain and the EP communication domain are 150 MB, respectively; and the service data flow in the PP communication domain is the smallest, and the communication buffer created for the PP communication domain is 50 MB.

[0095] In the distributed training business of the AI model, the communication performance of the business flow on different communication domains has different influences on the training iteration time consumption. The DP gradient summation business flow can be concurrent with the back calculation, and the back calculation operator does not depend on the completion of the gradient aggregation. The EP, SP, TP and PP business flows are executed in the front and back calculation, and the communication execution time directly affects the start time of the subsequent calculation operator. In summary, the priority of the EP, SP, TP and PP communication business flows should be higher than that of the DP business flow. Therefore, the embodiment of the present application provides a communication domain level buffer memory and a business flow priority configuration capability for the collection communication library, matches the communication resources according to the business flow traffic information, and optimizes the communication operator performance and memory occupation under the concurrent communication of multiple communication domains.

[0096] That is, when each communication domain is created, a priority parameter is configured for the communication connection of each communication domain, and the priority parameter indicates the scheduling priority of the collection operation of the business data flow of each communication domain when the collection operation of each communication domain is concurrently executed.

[0097] By configuring the priority of the communication connection of each communication domain, the communication domain level priority configuration capability is provided. When the collection operation of multiple communication domains is concurrently executed, the collection operation of multiple communication domains is scheduled according to the priority, more important business data flow collection operation is executed, and less important business data flow collection operation is scheduled, thereby optimizing the execution of the whole AI model training task.

[0098] In one example, the priority parameter of the communication connection of each communication domain is determined based on the dependency relationship between the operator of the parallel calculation task corresponding to each communication domain and the collection operation of each communication domain.

[0099] For example, the execution of the back calculation task has strong dependency on the collection operation of the EP communication domain, the execution of the back calculation task has strong dependency on the collection operation of the TP communication domain, and the execution of the back calculation task has strong dependency on the collection operation of the SP communication domain. The priority parameters of the communication connections of the EP communication domain, the TP communication domain and the SP communication domain are configured as high (high); the execution of the back calculation task has weak dependency on the communication operation of the PP communication domain, and the priority parameter of the communication connection of the PP communication domain is configured as middle (middle); the execution of the DP calculation task has no dependency on the collection operation of the DP communication domain, and the priority parameter of the communication connection of the DP communication domain is configured as low (low).

[0100] In step S503, the collection operation of the first type of communication domain and the collection operation of the second type of communication domain are concurrently executed.

[0101] The data transmission requests corresponding to each communication domain are generated based on the service data flow of the set communication operation to be performed by each communication domain in the first type of communication domain and the second type of communication domain. The scheduling proportion of the data transmission requests corresponding to each communication domain is determined based on the priority parameter of the communication connection of each communication domain. The target data transmission requests corresponding to each communication domain are scheduled based on the scheduling proportion. The target data transmission request corresponding to the first type of communication domain and the target data transmission request corresponding to the second type of communication domain are concurrently executed.

[0102] For example, the created communication domains include a DP communication domain, a TP communication domain, and an EP communication domain, the priority parameter of the communication connection of the DP communication domain is low, the priority parameter of the communication connection of the TP communication domain and the EP communication domain is high, and when the service flow of each communication domain in the DP communication domain, the TP communication domain, and the EP communication domain needs to perform the set communication operation, a plurality of data transmission requests are first generated according to the service data flow. For example, the communication service flow of the DP communication domain is 1 GB, but when the set communication operation is actually performed, the service data flow needs to be split into a plurality of data packets of 128B size for communication, and a data transmission request is generated for each data packet, so as to generate a plurality of data transmission requests. When the set communication operation of the DP communication domain and the set communication operation of the TP / EP communication domain are executed in parallel, the plurality of data transmission requests of each communication domain are executed according to the priority scheduling.

[0103] In the reverse calculation process, when the set communication operation of the DP communication domain and the set communication of the TP / EP communication domain are concurrently executed, the set communication operation requests of each communication domain are scheduled according to the priority of the communication connection of each communication domain. The high priority is scheduled more and the low priority is scheduled less. For example, the set communication operation of the DP communication domain and the set communication operation of the TP communication domain are concurrently executed, the DP communication domain generates 1000 data transmission requests, the TP communication domain generates 600 data transmission requests, the bandwidth of the physical communication link between NPU 0 and NPU 1 allows 600 data transmission tasks to be executed at a time, the scheduling proportion of the high priority and the low priority is 5:1, then 100 data transmission requests are scheduled from the 1000 data transmission requests of the DP communication domain, and 500 data transmission requests are scheduled from the 600 data transmission requests of the TP communication domain and are concurrently executed. In this way, the communication bandwidth of the physical link is filled, the bandwidth utilization of the physical communication link is improved, and the problem of mutual waiting between the communication connections of multiple communication domains in time is avoided, which causes the problem of insufficient calculation utilization.

[0104] FIG. 8 shows a signaling interaction diagram of distributed training of an AI model applying the collection communication method provided by the embodiments of the present application. As shown in FIG. 8, a business user submits a distributed training script of an AI model to a deep learning framework, which can parse the AI model according to the training script and perform distributed training on the AI model according to the model training logic indicated by the training script. Then the deep learning framework applies to a collection communication library for creating communication domains of each parallelism (DP, PP and EP) according to the cluster resources currently used by the business and the splitting manners of each parallelism (DP, PP and EP) given by the business user. The collection communication library creates independent communication connection resources and communication buffers for inter-device communication for each communication domain, and configures the service flow priority of the communication connection and the size of the communication buffer space according to the service flow characteristics of the communication domain. The deep learning framework issues communication operators of each parallelism (DP, PP and EP) to the collection communication library in each training iteration. The collection communication library matches the communication operators of each parallelism (DP, PP and EP) to the communication connection and communication buffer of the respective communication domain, and issues hardware execution. The DP communication domain operator has no data dependency relationship with the PP and EP communication domain operators, and can be executed concurrently according to the respective service flow priority. Finally, each communication domain communication operator feeds back the execution completion to the deep learning framework after the execution is completed.

[0105] The embodiments of the present application re-plan the scheduling and execution strategy of each communication domain on different parallelisms, create independent communication buffers and communication connections for each communication domain, realize concurrent scheduling and execution of the collection communication operations of each communication domain, and overcome the communication connection preemption problem when the collection communication operations of multiple communication domains are concurrently executed (see FIG. 9). Meanwhile, the embodiments of the present application provide communication buffer and service flow priority configuration capabilities at the communication domain level according to the traffic characteristics of different service flows in the distributed training of the AI model, realize the concurrent optimization of the reverse calculation and each parallel dimension communication in the complex parallel strategy combination scenario, and improve the utilization rate of the communication bandwidth of the physical communication link between each hardware accelerator.

[0106] Based on the same idea as the foregoing embodiment of the collection communication method, the embodiments of the present application also provide a collection communication device 1000, which can create independent communication buffers and communication connections for each communication domain in a plurality of communication domains that can be concurrently executed, realize concurrent execution of multiple communication domains, avoid communication connection preemption problems, and improve the execution efficiency of distributed computing tasks. The collection communication device 1000 includes units or modules to realize each step in the collection communication method shown in FIGS. 5-9.

[0107] FIG. 10 is a structural schematic diagram of a set communication device provided in an embodiment of the present application. As shown in FIG. 10, the set communication device 1000 is deployed in a distributed computing system, the distributed computing system includes N computing devices, N is a positive integer greater than 1, and the set communication device 100 provided in the present application at least includes an acquisition module 1001, a creation module 1002, and an execution module 1003, wherein the acquisition module 1001 is configured to acquire a to-be-computed task, the to-be-computed task includes a first type of parallel computing task and a second type of parallel computing task, and the first type of parallel computing task and the second type of parallel computing task have no dependency relationship; the creation module 1002 is configured to create a first type of communication domain and a second type of communication domain for the first type of parallel computing task and the second type of parallel computing task, respectively, wherein the first type of communication domain and the second type of communication domain at least include two same computing devices, each communication domain in the first type of communication domain and the second type of communication domain has an independent communication buffer and a communication connection, the first type of communication domain is configured to perform a set communication operation of a service data stream of the first type of parallel computing task, and the second type of communication domain is configured to perform a set communication operation of a service data stream of the second type of parallel computing task; and the execution module 1003 is configured to concurrently execute the set communication operation of the first type of communication domain and the set communication operation of the second type of communication domain.

[0108] In a possible implementation, the to-be-computed task is a distributed training task of an AI model; the first type of parallel computing task includes a data parallel computing task, and the second type of parallel computing task includes at least one of a pipeline parallel computing task, a tensor parallel computing task, a sequence parallel computing task, and an expert parallel computing task.

[0109] In another possible implementation, the N computing devices include N hardware accelerators, and the N hardware accelerators are distributed in one server or distributed in multiple servers.

[0110] In another possible implementation, each communication domain is determined based on a segmentation manner of M computing devices by a parallel computing task corresponding to each communication domain, the M computing devices are computing devices available for the distributed training task of the AI model in the N computing devices, and M is a positive integer less than or equal to N.

[0111] In another possible implementation, a size of a memory buffer of each communication domain is determined based on a size of a service data stream of a parallel computing task corresponding to each communication domain.

[0112] In another possible implementation, a communication connection of each communication domain is configured with a priority parameter, the priority parameter indicates a scheduling priority of a set operation of a service data stream of each communication domain when the set communication operation of each communication domain is concurrently executed.

[0113] In another possible implementation, the priority parameter of the communication connection of each communication domain is determined based on a dependency relationship between an operator of a parallel computing task corresponding to each communication domain and a collective communication operation of each communication domain.

[0114] In another possible implementation, the execution module 1003 is specifically configured to: generate a plurality of data transmission requests corresponding to each communication domain based on a service data stream on which each communication domain is to perform a collective communication operation; determine a scheduling ratio of the plurality of data transmission requests corresponding to each communication domain based on the priority parameter of the communication connection of each communication domain; schedule a target data transmission request corresponding to each communication domain based on the scheduling ratio; and concurrently execute the target data transmission request corresponding to the first type of communication domain and the target data transmission request corresponding to the second type of communication domain.

[0115] The collective communication apparatus 1000 according to the embodiments of the present application can correspond to the execution of the methods described in the embodiments of the present application, and the above and other operations and / or functions of each module in the collective communication apparatus 1000 are respectively used to implement the corresponding flow of each method in FIGS. 5-9, and for brevity, will not be repeated here.

[0116] The embodiments of the present application also provide a computing device, including at least one processor, a memory and a communication interface, the processor being configured to execute the methods described in FIGS. 5-9.

[0117] FIG. 11 is a structural schematic diagram of a computing device according to an embodiment of the present application.

[0118] As shown in FIG. 11, the computing device 1100 includes at least one processor 1101, a memory 1102 and a communication interface 1103. The processor 1101, the memory 1102 and the communication interface 1103 are communicatively connected, and can be communicatively connected in a wired (for example, bus) manner or in a wireless manner. The communication interface 1103 is configured to send and / or receive data sent by other devices; the memory 1102 stores computer instructions, and the processor 1101 executes the computer instructions to execute the method in the foregoing method embodiments, so as to create an independent communication buffer and a communication connection for each communication domain in a plurality of communication domains that can be concurrently executed, implement concurrent execution of the plurality of communication domains, avoid communication connection preemption, and improve the execution efficiency of a distributed computing task.

[0119] It should be understood that, in the embodiments of the present application, the processor 1101 can be a central processing unit CPU, and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0120] The memory 1102 can include a read-only memory and a random access memory, and provide instructions and data for the processor 1101. The memory 1102 can also include a non-volatile random access memory. Optionally, the random access memory can be a high bandwidth memory (HBM), for example.

[0121] The memory 1102 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0122] It should be understood that the computing device 1100 according to the embodiments of the present application can perform the method shown in FIGS. 5-9, the detailed description of which is described above, and for brevity, will not be repeated here.

[0123] The embodiments of the present application provide a computer readable storage medium, which stores a computer program, when the computer program is executed by a processor, the above-mentioned method is implemented.

[0124] The embodiments of the present application provide a chip, which comprises at least one processor and an interface, the at least one processor determines program instructions or data through the interface; the at least one processor is used to execute the program instructions, so as to implement the above-mentioned method.

[0125] The embodiments of the present application provide a computer program or computer program product, which comprises instructions, when the instructions are executed, the computer executes the above-mentioned method.

[0126] Those skilled in the art should further appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0127] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be implemented in hardware, software executed by a processor, or a combination of both. The software module can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0128] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A collective communication method, characterized in that, Applied to a distributed computing system, the distributed computing system comprising N computing devices, where N is a positive integer greater than 1, the method includes: Obtain the task to be computed, which includes a first type of parallel computing task and a second type of parallel computing task; A first type of communication domain and a second type of communication domain are created for the first type of parallel computing task and the second type of parallel computing task, respectively. At least some of the communication domains in the first type of communication domain and at least some of the communication domains in the second type of communication domain have independent communication buffers and communication connections. The first type of communication domain is used to perform the collection communication operation of the business data stream of the first type of parallel computing task, and the second type of communication domain is used to perform the collection communication operation of the business data stream of the second type of parallel computing task. Concurrently execute the collection communication operations of the first type of communication domain and the collection communication operations of the second type of communication domain.

2. The method according to claim 1, characterized in that, The task to be computed is a distributed training task of an AI model; the first type of parallel computing task includes data parallel computing tasks, and the second type of parallel computing task includes at least one of pipeline parallel computing tasks, tensor parallel computing tasks, sequence parallel computing tasks, and expert parallel computing tasks.

3. The method according to claim 2, characterized in that, The N computing devices include N hardware accelerators, which are distributed across one server or across multiple servers.

4. The method according to any one of claims 2 or 3, characterized in that, Each communication domain is determined based on the method of partitioning the M computing devices according to the parallel computing tasks corresponding to each communication domain. The M computing devices are the computing devices available for the distributed training tasks of the AI ​​model among the N computing devices, and M is a positive integer less than or equal to N.

5. The method according to any one of claims 1-4, characterized in that, The size of the communication buffer for each communication domain is determined based on the size of the business data stream of the parallel computing task corresponding to each communication domain.

6. The method according to any one of claims 1-5, characterized in that, The communication connections of each communication domain are configured with priority parameters, which indicate the scheduling priority of the collective communication operations of the service data streams of each communication domain when the collective communication operations of each communication domain are executed concurrently.

7. The method according to claim 6, characterized in that, The priority parameters of the communication connections in each communication domain are determined based on the dependency relationship between the collective communication operations of each communication domain and the operators computed in reverse in the distributed training task of the AI ​​model.

8. The method according to claim 6 or 7, characterized in that, The concurrent execution of the collective communication operations of the first type of communication domain and the collective communication operations of the second type of communication domain includes: Based on the business data streams of the collective communication operations to be performed in each communication domain, multiple data transmission requests corresponding to each communication domain are generated. Based on the priority parameters of the communication connections in each communication domain, the scheduling ratio of multiple data transmission requests corresponding to each communication domain is determined. Based on the scheduling ratio, the target data transmission requests corresponding to each communication domain are scheduled. The scheduling is executed concurrently to obtain the target data transmission request corresponding to the first type of communication domain and the target data transmission request corresponding to the second type of communication domain.

9. The method according to claim 1, characterized in that, The first type of communication domain and the second type of communication domain each include at least two identical computing devices.

10. A collective communication device, characterized in that, Deployed in a distributed computing system, the distributed computing system comprising N computing devices, where N is a positive integer greater than 1, the device comprising: The acquisition module is used to acquire tasks to be computed, which include a first type of parallel computing task and a second type of parallel computing task. A creation module is used to create a first type of communication domain and a second type of communication domain for the first type of parallel computing task and the second type of parallel computing task, respectively. At least some of the communication domains in the first type of communication domain and at least some of the communication domains in the second type of communication domain have independent communication buffers and communication connections. The first type of communication domain is used to perform the collection communication operation of the business data stream of the first type of parallel computing task, and the second type of communication domain is used to perform the collection communication operation of the business data stream of the second type of parallel computing task. An execution module is used to concurrently execute the collective communication operations of the first type of communication domain and the collective communication operations of the second type of communication domain.

11. The aggregated communication device according to claim 10, characterized in that, The task to be computed is a distributed training task of an AI model; the first type of parallel computing task includes data parallel computing tasks, and the second type of parallel computing task includes at least one of pipeline parallel computing tasks, tensor parallel computing tasks, sequence parallel computing tasks, and expert parallel computing tasks.

12. The aggregated communication device according to claim 10 or 11, characterized in that, The N computing devices include N hardware accelerators, which are distributed across one server or across multiple servers.

13. The collective communication device according to any one of claims 10-12, characterized in that, Each communication domain is determined based on the method of partitioning the M computing devices according to the parallel computing tasks corresponding to each communication domain. The M computing devices are the computing devices available for the distributed training tasks of the AI ​​model among the N computing devices, and M is a positive integer less than or equal to N.

14. The collective communication device according to any one of claims 10-13, characterized in that, The size of the communication buffer for each communication domain is determined based on the size of the business data stream of the parallel computing task corresponding to each communication domain.

15. The collective communication device according to any one of claims 10-14, characterized in that, The communication connections of each communication domain are configured with priority parameters, which indicate the scheduling priority of the collective communication operations of the service data streams of each communication domain when the collective communication operations of each communication domain are executed concurrently.

16. The aggregated communication device according to claim 15, characterized in that, The priority parameters of the communication connections in each communication domain are determined based on the dependency relationship between the collective communication operations of each communication domain and the operators computed in reverse in the distributed training task of the AI ​​model.

17. A computing device, comprising a memory and a processor, characterized in that, The memory stores instructions that, when executed by a processor, cause the method described in any one of claims 1-9 to be implemented.

Images