GPU communication method and device, storage medium and program product

Abstract

The invention provides a GPU communication method and device, a storage medium and a program product, and relates to the technical field of artificial intelligence large language model training, and the method comprises the steps: obtaining communication request information; the communication request information is used for requesting to forward transmission data of the source end GPU to the destination end GPU; determining equipment types of the source end GPU and the destination end GPU based on the communication request information; determining a target communication library based on the equipment types of the source end GPU and the destination end GPU; and forwarding the transmission data of the source end GPU to the destination end GPU through the target communication library. The method is used for improving the communication efficiency between the GPUs.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence large language model training, and particularly relates to a GPU communication method, device, storage medium and program product. BACKGROUND

[0002] In recent years, with the rapid development of the hardware ecology of graphics processing units (GPUs), multiple GPU brands and architecture types have emerged on the market. The diversification of GPUs provides more optional hardware resources for large language model training. In particular, in scenarios where computing power resources are scarce or cost-effectiveness is pursued, heterogeneous GPU deployment has become a more and more common and realistic choice. Distributed training of large language models can be achieved by deploying computing nodes of heterogeneous GPUs.

[0003] However, the communication efficiency between GPUs in the current distributed training process needs to be improved. SUMMARY

[0004] Therefore, the present disclosure provides a GPU communication method, device, storage medium and program product, which can select a corresponding target communication library based on the device types of source and destination GPUs in the communication process to forward the transmission data, save the communication overhead when forwarding through the CPU, and improve the communication efficiency between GPUs.

[0005] In a first aspect, the present disclosure provides a GPU communication method, which is applied to a computing node, and the method comprises: obtaining communication request information; the communication request information is used to request to forward transmission data of a source GPU to a destination GPU; determining the device types of the source and destination GPUs based on the communication request information; determining a target communication library based on the device types of the source and destination GPUs; and forwarding the transmission data of the source GPU to the destination GPU through the target communication library.

[0006] The GPU communication method provided by the present disclosure can determine the device types of the source and destination GPUs based on the communication request information, determine a target communication library based on the device types of the source and destination GPUs, and forward the transmission data of the source GPU to the destination GPU through the target communication library. In this way, compared with the transit mechanism in which the CPU is involved in all GPU communications in the related art, the present disclosure can select to use the target communication library to realize the communication between GPUs, save the communication overhead when forwarding through the CPU, and improve the communication efficiency between GPUs.

[0007] In a second aspect, the present disclosure provides a GPU communication device, which comprises various functional modules for the method described in the first aspect.

[0008] In a third aspect, the present disclosure provides an electronic device, comprising: a plurality of GPUs, further comprising: a processor and a memory; the memory stores instructions executable by the processor; the processor is configured to execute the foregoing instructions, so that the electronic device implements the method of the first aspect.

[0009] In a fourth aspect, the present disclosure provides a readable storage medium, comprising: software instructions; when the software instructions are run in an electronic device, the electronic device implements the method of the first aspect.

[0010] In a fifth aspect, the present disclosure provides a computer program product, comprising: computer instructions; when the computer instructions are run in an electronic device, the electronic device implements the method of the first aspect.

[0011] The advantages of the second aspect to the fifth aspect above can be referred to the first aspect, and will not be repeated. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings are included to provide a further understanding of the technical scheme of the present application, and constitute a part of the specification, and are used together with the embodiments of the present disclosure to explain the technical scheme of the present application, and do not constitute a limitation on the technical scheme of the present application.

[0013] Figure 1 The figure is a schematic diagram of the communication process between GPUs under the PP architecture;

[0014] Figure 2 The figure is a schematic diagram of the architecture of the GPU communication system provided by the embodiment of the present disclosure;

[0015] Figure 3 The figure is a schematic diagram of the composition of the patch package module provided by the embodiment of the present disclosure;

[0016] Figure 4 The figure is a schematic diagram of the GPU communication method provided by the embodiment of the present disclosure;

[0017] Figure 5 The figure is a schematic diagram of the homogeneous communication provided by the embodiment of the present disclosure;

[0018] Figure 6 The figure is a schematic diagram of the heterogeneous communication provided by the embodiment of the present disclosure;

[0019] Figure 7 The figure is a schematic diagram of the GPU communication optimization provided by the embodiment of the present disclosure;

[0020] Figure 8 The figure is a schematic diagram of the composition of the GPU communication device provided by the embodiment of the present disclosure;

[0021] Figure 9A constituent schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present disclosure.

[0023] Unless otherwise required by context, the term “comprise” and other forms such as “comprises”, “comprises”, and “comprising” are to be construed as open, inclusive, meaning that “comprising” means “including, but not limited to”. In the description of the specification, the terms “one embodiment”, “some embodiments”, “exemplary embodiments”, “example”, “specific example” or “some examples” and the like are intended to mean that a particular feature, structure, material or characteristic included in at least one embodiment or example of the present disclosure. The illustrative representation of the above terms does not necessarily mean the same embodiment or example. In addition, the specific features, structures, materials or characteristics described can be included in any one or more embodiments or examples in any appropriate manner.

[0024] The terms “first”, “second” are only used for description purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with “first”, “second” can explicitly or implicitly include one or more features. In the description of the present disclosure, unless otherwise stated, the meaning of “multiple” is two or more.

[0025] In the embodiments of the present disclosure, the words “exemplarily” or “for example” are used to mean as an example, illustration or description. Any embodiment or design scheme described as “exemplarily” or “for example” in the embodiments of the present disclosure should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words “exemplarily” or “for example” are intended to present the relevant concept in a specific manner.

[0026] In addition, the use of "based on" means openness and inclusivity, as a process, step, calculation, or other action "based on" one or more stated conditions or values can in practice be based on additional conditions or values beyond those stated.

[0027] In recent years, with the rapid development of the hardware ecology of Graphics Processing Units (GPUs), multiple GPU brands and architecture types have emerged on the market. The diversification of GPUs provides more optional hardware resources for large language model training, especially in scenarios where computing resources are scarce or cost-effectiveness is pursued. Heterogeneous GPU deployment has become a more and more common and realistic choice. Distributed training of large language models can be achieved by deploying computing nodes with heterogeneous GPUs.

[0028] In the distributed training of large language models, Pipeline Parallelism (PP) is a key parallel strategy. It divides a large language model into several stages according to layers, and each stage is carried out by one or more GPUs, thereby breaking through the memory limit of a single card and enabling the training of super-large models.

[0029] Exemplarily, Figure 1 is a schematic diagram of the communication process between GPUs under the PP architecture. As Figure 1 shown, taking eight GPUs with identifiers 0, 2, 4, 6, 8, 10, 12, and 14 as examples, under the PP architecture, the eight GPUs are connected in series, and data can only be transmitted between adjacent GPUs each time data is transmitted. Taking the transmission of data from the GPU with identifier 0 to the GPU with identifier 14 as an example, the data transmission path is 0-2-4-6-8-10-12-14 in turn. In each data transmission, a Central Processing Unit (CPU) is used for forwarding. Specifically, the data of the source GPU is first copied to the CPU memory of the source GPU, forwarded to the destination CPU via the network or local shared memory, and then written to the target GPU.

[0030] From the analysis in the above Figure 1 embodiment, although PP has good adaptability in a heterogeneous environment, the communication efficiency between GPUs in the existing PP strategy needs to be improved.

[0031] Based on this, the embodiments of the present disclosure provide a GPU communication method, device, storage medium, and program product. The method can select a corresponding target communication library to forward and transmit data based on the device types of the source GPU and the destination GPU in the communication process, save the communication overhead when forwarding through the CPU, and improve the communication efficiency between GPUs.

[0032] The following is an introduction with reference to the accompanying drawings.

[0033] Figure 2 This is a schematic diagram of the architecture of the GPU communication system provided by the embodiment of the present disclosure. Figure 2 As shown, the system includes: a training framework layer, a pytorch layer, a patch package module, a communication backend layer, and a hardware layer.

[0034] The training framework layer can be used to define model parallel strategies (such as the PP strategy mentioned above), split large language models into multiple GPUs in the hardware layer for training, and call the PyTorch layer to initialize the communication backend layer.

[0035] As an example, Figure 2 As shown, the training framework layer can be specifically implemented as a Megatron framework.

[0036] The pytorch layer can be used to perform tensor computations on the GPU and calculate gradients, which are synchronized via the communication library in the communication backend layer.

[0037] The communication backend layer can include multiple communication libraries, and different GPU device types can correspond to different communication libraries. For example, the device type can specifically be the GPU device model or manufacturer, and each device model or manufacturer can correspond to a communication library.

[0038] The hardware layer can include multiple GPUs to provide computing power support for large language model training.

[0039] The patch package module can be connected to the pytorch layer for communication. The patch package module can be specifically used to select the corresponding target communication library to forward transmission data based on the device types of the source GPU and the destination GPU during the communication process. The specific process can be referred to the GPU communication method provided in the following embodiment, which will not be repeated here.

[0040] In some possible embodiments, Figure 3 This is a schematic diagram of the composition of the patch package module provided in the embodiment of the present disclosure. Figure 3 As shown, the pytorch layer may specifically include a first communication interface. The patch package module may specifically include: a second communication interface, a device type analysis module, a path decision module, a communication task distribution module, a homogeneous communication processing module, and a heterogeneous communication processing module.

[0041] The model training framework layer can generate communication request information and send the communication request information to the PyTorch layer through the first communication interface; the communication request information is used to request that the transmission data of the source GPU be forwarded to the destination GPU.

[0042] The second communication interface is connected with the first communication interface. The second communication interface can be used to intercept the communication request information from the first communication interface.

[0043] The device type analysis module can be used to determine the device types of the source GPU and the destination GPU based on the communication request information.

[0044] The path decision module can be used to determine the target communication library based on the device types of the source GPU and the destination GPU.

[0045] The communication task distribution module can be used to generate a communication task based on the target communication library and distribute the communication task to the homogeneous communication processing module or the heterogeneous communication processing module.

[0046] For example, in the case where the device types of the source GPU and the destination GPU are the same, the communication task distribution module can generate a homogeneous communication task and send the homogeneous communication task to the homogeneous communication processing module, so that the homogeneous communication processing module calls the target communication library to forward the transmission data of the source GPU to the destination GPU.

[0047] For another example, in the case where the device types of the source GPU and the destination GPU are different, the communication task distribution module can generate a heterogeneous communication task and send the heterogeneous communication task to the heterogeneous communication processing module, so that the heterogeneous communication processing module calls the target communication library to forward the transmission data of the source GPU to the destination GPU.

[0048] The GPU communication method provided by the embodiments of the present disclosure is executed by a GPU communication device. The GPU communication device can be a computing node as described above, or a processor (such as a CPU) in the computing node, or a software system or platform installed in the computing node for executing the GPU communication method, or a functional module (such as the patch module described above) in the computing node for executing the GPU communication method. The embodiments of the present disclosure do not limit this. Figure 2 The GPU communication device can be a computing node as described above, or a processor (such as a CPU) in the computing node, or a software system or platform installed in the computing node for executing the GPU communication method, or a functional module (such as the patch module described above) in the computing node for executing the GPU communication method. The embodiments of the present disclosure do not limit this.

[0049] For the sake of simplicity, the following will be introduced by taking the GPU communication device as the computing node as an example.

[0050] Figure 4 A flowchart of the GPU communication method provided by the embodiments of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the method can include the following steps: Figure 4

[0051] S101, obtaining communication request information.

[0052] The communication request information is used to request to forward the transmission data of the source GPU to the destination GPU. ​

[0053] As an example, as described above, the training framework layer (e.g., the Megatron framework) in the computing node can split the large language model into multiple GPUs for training, the training framework layer can schedule data transmission in the training process, in this case, the communication request information can be generated by the training framework layer and sent to the pytorch layer through the first communication interface, and the computing node can specifically intercept the communication request information through the second communication interface in the patch package module.

[0054] S102, based on the communication request information, determining the device types of the source GPU and the destination GPU.

[0055] In a possible implementation, the training framework layer can directly add the device types of the source GPU and the destination GPU to the communication request information when generating the communication request information. In this case, the computing node can directly read the communication request information to determine the device types of the source GPU and the destination GPU.

[0056] In another possible implementation, the computing node can prestore a device type correspondence relationship, which can be used to represent the correspondence between the identifier of the GPU and the device type, and the training framework layer can add the identifier of the source GPU and the identifier of the destination GPU to the generated communication request information. In this case, the computing node can determine the device types of the source GPU and the destination GPU based on the identifier of the source GPU, the identifier of the destination GPU, and the preset device type correspondence relationship.

[0057] As an example, the computing node can take the identifier of the source GPU as an index to search the device type correspondence relationship, take the device type corresponding to the identifier of the source GPU in the device type correspondence relationship as the device type of the source GPU, take the identifier of the destination GPU as an index to search the device type correspondence relationship, and take the device type corresponding to the identifier of the destination GPU in the device type correspondence relationship as the device type of the destination GPU.

[0058] Exemplarily, the device type correspondence relationship can be represented as Table 1 as follows:

[0059] Table 1

[0060] GPU identification Device type Identification 1 Type 1 Identification 2 Type 2 Identification 3 Type 3

[0061] As shown in Table 1, the device type correspondence relationship can specifically include a GPU identifier item and a device type item. The GPU identifier item includes identifier 1, identifier 2, and identifier 3; the device type item includes type 1, type 2, and type 3. Identifier 1 and type 1 have a corresponding relationship; identifier 2 and type 2 have a corresponding relationship; and identifier 3 and type 3 have a corresponding relationship.

[0062] S103, determining a target communication library based on the device type of the source GPU and the device type of the destination GPU.

[0063] In a possible implementation, when the device type of the source GPU and the device type of the destination GPU are both target device types, the computing node can determine a target communication library corresponding to the target device types based on the target device types and a preset communication library correspondence.

[0064] The communication library correspondence is used to represent the correspondence between the device type and the communication library. In order to distinguish from the device type correspondence in Table 1, the communication library correspondence can also be referred to as a first correspondence, and the device type correspondence can also be referred to as a second correspondence. The embodiments of the present disclosure do not limit this.

[0065] As an example, the computing node can take the target device type as an index to traverse the communication library correspondence, and take the communication library corresponding to the target device type in the communication library correspondence as the target communication library.

[0066] Exemplarily, the communication library correspondence can be as shown in Table 2:

[0067] Table 2

[0068] Device type Communication library Type 1 Communication library 1 Type 2 Communication library 2 Type 3 Communication library 3

[0069] As shown in Table 2, the communication library correspondence can include a device type item and a communication library item. The device type item includes type 1, type 2, and type 3. The communication library item includes communication library 1, communication library 2, and communication library 3. Type 1 and communication library 1 have a correspondence relationship. Type 2 and communication library 2 have a correspondence relationship. Type 3 and communication library 3 have a correspondence relationship.

[0070] S104, forwarding the transmission data of the source GPU to the destination GPU through the target communication library.

[0071] The target communication library can include the underlying components and execution logic required to forward the transmission data between GPUs. The specific process of forwarding the transmission data between GPUs by the computing node through the target communication library can refer to the related art, which will not be described here.

[0072] The GPU communication method provided by the embodiments of the present disclosure can determine the device types of the source GPU and the destination GPU based on the communication request information, determine the target communication library based on the device types of the source GPU and the destination GPU, and forward the transmission data of the source GPU to the destination GPU through the target communication library. In this way, compared with the transit mechanism in the related art in which all GPU communications need to involve the CPU, the present disclosure can select to use the target communication library to implement the communication between GPUs, save the communication overhead when forwarding through the CPU, and improve the communication efficiency between GPUs.

[0073] Meanwhile, thanks to the improved update efficiency between GPUs, the end-to-end training speed can be greatly improved, and the training performance advantage of the present disclosure is more obvious as the size of the large language model increases.

[0074] In addition, the present disclosure also reduces the participation range of the CPU in the GPU communication, and significantly reduces the resource occupation of the CPU.

[0075] In some possible embodiments, before determining the target communication library corresponding to the target type, the computing node can further determine whether the device type of the source GPU and the device type of the destination GPU are the same.

[0076] As an example, as described above, the computing node can determine the device type of the source GPU and the device type of the destination GPU based on the above-mentioned device type correspondence (second correspondence) respectively. In this case, the computing node can directly determine whether the device type of the source GPU and the device type of the destination GPU are the same.

[0077] As another example, the computing node can include a plurality of (pre-divided) homogeneous communication groups, the device types of the GPUs in one homogeneous communication group are the same, and the communication request information can include the identifier of the source GPU and the identifier of the destination GPU. In this case, the computing node can determine whether the device type of the source GPU and the device type of the destination GPU are the same based on the identifier of the source GPU, the identifier of the destination GPU, and the identifier of the GPU in the homogeneous communication group.

[0078] For example, the computing node can determine that the device type of the source GPU and the device type of the destination GPU are both the target device type corresponding to the target homogeneous communication group in response to the identifier of the source GPU and the identifier of the destination GPU being the identifier in the target homogeneous communication group.

[0079] Exemplarily, Figure 5 A homogeneous communication schematic diagram is provided for the embodiments of the present disclosure. As Figure 5As shown, taking eight GPUs with IDs of 0, 2, 4, 6, 8, 10, 12, and 14 as an example, assuming that the device types of the GPUs with IDs of 0, 2, 4, and 6 are Type 1, and the device types of the GPUs with IDs of 8, 10, 12, and 14 are Type 2, the GPUs with IDs of 0, 2, 4, and 6 can be divided into one homogeneous communication group 1, and the GPUs with IDs of 8, 10, 12, and 14 can be divided into one homogeneous communication group 2. The GPUs in the homogeneous communication group 1 can communicate using the corresponding communication library 1, and the GPUs in the homogeneous communication group 2 can communicate using the corresponding communication library 2.

[0080] For another example, the plurality of GPUs in the computing node can be GPUs of the PP architecture, and the computing node can determine that the device type of the source-end GPU and the device type of the destination-end GPU are different in response to at least one of the following conditions being met by the ID of the source-end GPU or the ID of the destination-end GPU:

[0081] (1) The ID of the source-end GPU and the ID of the destination-end GPU are the IDs of GPUs in different homogeneous communication groups, respectively.

[0082] (2) The ID of the source-end GPU is the ID of the last GPU in any one of the homogeneous communication groups.

[0083] (3) The ID of the destination-end GPU is the ID of the first GPU in any one of the homogeneous communication groups.

[0084] Exemplarily, Figure 6 A heterogeneous communication schematic diagram is provided for the embodiments of the present disclosure. As shown in FIG. 1, the computing node includes a plurality of GPUs, and the plurality of GPUs can be divided into a plurality of homogeneous communication groups. The GPUs in each of the homogeneous communication groups can communicate using a corresponding communication library. Figure 6As shown, for example, the GPUs with the above identifiers 0, 2, 4, and 6 are divided into one homogeneous communication group 1, and the GPUs with the above identifiers 8, 10, 12, and 14 are divided into one homogeneous communication group 2. Assuming that the identifier of the source GPU in the current communication request information is 6, and the identifier of the destination GPU is 8, the computing node can determine that the device types of the source GPU and the destination GPU are different according to that the identifier 6 is the identifier of the GPU in the homogeneous communication group 1, the identifier 8 is the identifier of the GPU in the homogeneous communication group 2, and the GPU with the identifier 6 and the GPU with the identifier 8 are GPUs in different homogeneous communication groups. Alternatively, the computing node can determine that the device types of the source GPU and the destination GPU are different according to that the identifier 6 is the identifier of the last GPU in the homogeneous communication group 1, and the source GPU must communicate with the GPUs in other homogeneous communication groups according to the PP architecture, that is, the destination GPU must be a GPU in other homogeneous communication groups. Alternatively, the computing node can determine that the device types of the source GPU and the destination GPU are different according to that the identifier 8 is the identifier of the first GPU in the homogeneous communication group 2, and the destination GPU must be a GPU in other homogeneous communication groups according to the PP architecture, that is, the source GPU must be a GPU in other homogeneous communication groups. The source GPU and the destination GPU can forward the transmission data through the transit GPU.

[0085] In some embodiments, in the case where the device type of the source GPU and the device type of the destination GPU are different, the computing node can forward the transmission data of the source GPU to the destination GPU through the transit CPU.

[0086] As an example, the computing node can specifically forward the transmission data of the source GPU to the destination GPU through the transit CPU and a Cross-device Comm Lib (XCCL).

[0087] The XCCL can specifically include a communication enhancement component and a memory pool management component.

[0088] For example, for the communication enhancement component of the XCCL, the transit CPU can include a source CPU and a destination CPU; the source CPU is in communication connection with the source GPU; and the destination GPU is in communication connection with the destination GPU. In this case, the computing node can copy the transmission data of the source GPU to a Remote Direct Memory Access (RDMA) buffer of the source CPU, call an RDMA interface to transmit the transmission data from the RDMA buffer of the source CPU to an RDMA buffer of the destination CPU, and copy the transmission data from the RDMA buffer of the destination CPU to the destination GPU.

[0089] For example, for the memory pool management component part of XCCL, XCCL can maintain a pre-registered buffer (Memory Region, MR) pool, which includes MRs of various sizes (such as 32MB or 256MB, etc.), and dynamically allocates and recycles according to the PP tensor scale. For example, for a 500MB activation tensor, XCCL can allocate a 512 MR from the MR pool, and after the transmission is completed, it is reused for the next communication multiplexing. XCCL can also predict MR requirements according to PP communication group tensor metadata, for example, high-frequency small tensors (100MB gradient fragments) are preferentially allocated 128MB MR, and low-frequency large tensors (such as 8GB activation values) are allocated 1GB MR, and idle MR is dynamically recycled.

[0090] It should be understood that in the related art, when forwarding the transmission data of the GPU through the CPU relay mechanism, the transmission control protocol (Transmission Control Protocol, TCP) mechanism is usually used for transmission. The specific transmission path is: source GPU-source CPU user mode memory-source CPU kernel mode TCP buffer-destination CPU kernel mode TCP buffer-destination CPU user mode memory-destination GPU. As can be seen, the TCP transmission mechanism involves multiple user mode / kernel mode switches, and each switch will bring copy overhead. The GPU communication method provided by the embodiment of the disclosure can copy the transmission data of the source GPU to the remote direct memory access (RDMA) buffer of the source CPU, call the RDMA interface to transmit the transmission data from the RDMA buffer of the source CPU to the RDMA buffer of the destination CPU, and copy the transmission data from the RDMA buffer of the destination CPU to the destination GPU, in the case where the device type of the source GPU and the device type of the destination GPU are different. The communication can be directly performed through the RDMA buffer, eliminating the multiple user mode / kernel mode switches in the middle, saving the copy overhead brought by each switch, reducing the delay from milliseconds to microseconds, and the bandwidth is close to the limit of the network card.

[0091] Based on the understanding of the above embodiments, Figure 7 The GPU communication optimization schematic diagram provided by the embodiment of the disclosure is shown in FIG. 1. Figure 7 As shown, taking the Megatron framework as an example, the optimization process can include the following steps:

[0092] S201, the Megatron framework is initialized.

[0093] S202, the pytorch layer initializes the communication group.

[0094] S203, create the original PP communication group.

[0095] As an example, two PP communication groups are created with 16 GPUs, the computing node can create PP communication group 1 and PP communication group 2, the GPU identification in PP communication group 1 is: {0, 2, 4, 6, 8, 10, 12, 14}, and the GPU identification in PP communication group 2 is: {1, 3, 5, 7, 9, 11, 13, 15}.

[0096] S204, further divide the homogeneous communication group and the heterogeneous communication group, and bind a communication library for each homogeneous communication group and heterogeneous communication group.

[0097] The device types of the GPUs in the homogeneous communication group are the same.

[0098] For example, the GPUs identified as {0, 2, 4, 6} in PP communication group 1 can be divided into a homogeneous communication group 1, and the group communication can use communication library 1. The GPUs identified as {8, 10, 12, 14} in PP communication group 1 can be divided into a homogeneous communication group 2, and the group communication can use communication library 2.

[0099] For another example, the GPUs identified as {0, 2, 4, 6, 8, 10, 12, 14} in PP communication group 1 can be divided into a heterogeneous communication group, and the group communication can use XCCL.

[0100] S205, initiating PP communication in the large language model training process.

[0101] S206, obtaining the device types of the GPUs of the communication parties.

[0102] S207, judging whether the communication group used is a homogeneous GPU according to the device types of the GPUs of the communication parties.

[0103] If yes, S208 is executed; if no, S209 is executed.

[0104] S208, selecting the communication library bound by the corresponding homogeneous communication group to communicate.

[0105] S209, communicating through the forwarding of the relay CPU.

[0106] The above describes the solutions provided by the embodiments of the present disclosure from the perspective of the method. In order to implement the above functions, each device, for example, the computing node, comprises a hardware structure and / or a software module for executing each function. The skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. The skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0107] The embodiments of the present disclosure can divide the computing node into functional modules according to the above method embodiments. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one functional module. The above integrated module can be implemented in the form of hardware or software. It should be noted that the division of the module in the embodiments of the present disclosure is illustrative, and is only a logical function division. In actual implementation, there can be another division manner. The following takes the example of dividing each functional module according to each function.

[0108] In an exemplary embodiment, the present disclosure provides a GPU communication device, which can be applied to the above computing node. Figure 8 The composition schematic diagram of the GPU communication device provided by the embodiments of the present disclosure is shown in FIG. 8. As shown in FIG. 8, the device comprises an acquisition module 801 and a processing module 802. Figure 8 The acquisition module 801 is configured to acquire communication request information, and the communication request information is used to request to forward the transmission data of the source GPU to the destination GPU.

[0109] The acquisition module 801 is configured to acquire communication request information, and the communication request information is used to request to forward the transmission data of the source GPU to the destination GPU.

[0110] The processing module 802 is configured to determine the device types of the source GPU and the destination GPU based on the communication request information, determine the target communication library based on the device types of the source GPU and the destination GPU, and forward the transmission data of the source GPU to the destination GPU through the target communication library.

[0111] In some possible embodiments, the processing module 802 is specifically configured to, in the case that the device type of the source GPU and the device type of the destination GPU are both the target device type, determine the target communication library corresponding to the target device type based on the target device type and a preset first correspondence relationship, and the first correspondence relationship is used to represent the correspondence relationship between the device type and the communication library.

[0112] In some possible embodiments, the communication request information comprises an identifier of the source GPU and an identifier of the destination GPU; and the processing module 802 is specifically configured to determine the device type of the source GPU and the device type of the destination GPU based on the identifier of the source GPU, the identifier of the destination GPU, and a preset second correspondence relationship, wherein the second correspondence relationship is used to represent a correspondence relationship between the identifier of the GPU and the device type.

[0113] In some possible embodiments, the computing node comprises a plurality of homogeneous communication groups, and the device types of the GPUs in one homogeneous communication group are the same; the communication request information comprises an identifier of the source GPU and an identifier of the destination GPU; and the processing module 802 is further configured to determine that the device type of the source GPU and the device type of the destination GPU are both a target device type corresponding to a target homogeneous communication group, in response to the identifier of the source GPU and the identifier of the destination GPU being identifiers in the target homogeneous communication group.

[0114] In some possible embodiments, the processing module 802 is further configured to determine that the device type of the source GPU and the device type of the destination GPU are different, in response to at least one of the following being met: the identifier of the source GPU and the identifier of the destination GPU being identifiers of GPUs in different homogeneous communication groups respectively; the identifier of the source GPU being an identifier of a last GPU in any one of the homogeneous communication groups; and the identifier of the destination GPU being an identifier of a first GPU in any one of the homogeneous communication groups.

[0115] In some possible embodiments, the processing module 802 is further configured to forward, by the transit CPU, the transmission data of the source GPU to the destination GPU, in a case where the device type of the source GPU and the device type of the destination GPU are different.

[0116] In some possible embodiments, the transit CPU comprises a source CPU and a destination CPU; the source CPU is in communication connection with the source GPU; and the destination GPU is in communication connection with the destination GPU; and the processing module 802 is specifically configured to copy the transmission data of the source GPU to a remote direct memory access (RDMA) buffer of the source CPU; call an RDMA interface to transmit the transmission data from the RDMA buffer of the source CPU to an RDMA buffer of the destination CPU; and copy the transmission data from the RDMA buffer of the destination CPU to the destination GPU.

[0117] It should be noted that the modules in the above description can also be referred to as units, for example, the processing module can be referred to as a processing unit. In addition, in the embodiments shown in the above description, the names of the modules can also be different from those shown in the figures, for example, the acquisition module can also be referred to as a transceiving module or a communication module, and the like. Figure 8 Figure 8 In some possible embodiments, the processing module 802 is specifically configured to determine the device type of the source GPU and the device type of the destination GPU based on the identifier of the source GPU, the identifier of the destination GPU, and a preset second correspondence relationship, wherein the second correspondence relationship is used to represent a correspondence relationship between the identifier of the GPU and the device type. ​

[0118] Figure 8 Each module in the above-mentioned modules, if implemented in the form of a software function module and sold or used as an independent product, can be stored in a readable storage medium. Based on such an understanding, the technical solutions of the embodiments of the present disclosure essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to make an electronic device (which can be a mobile phone, a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the methods of the embodiments of the present disclosure. The storage medium storing the computer software product includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0119] In the case of the above-mentioned CPU communication device adopting the form of hardware to realize the functions of the above-mentioned integrated modules, the embodiments of the present disclosure further provide an electronic device. Figure 9 The electronic device provided by the embodiments of the present disclosure is shown in the form of a composition diagram. As shown in the figure, the electronic device includes a processor 902, a communication interface 903, and a bus 904. As an example, the electronic device can also include a memory 901. As an example, the electronic device can also include a GPU 905, which can include multiple GPUs. Figure 9

[0120] The processor 902 can be various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 902 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, transistor logic device, hardware component or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 902 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0121] The communication interface 903 is used to connect with other devices through a communication network. The communication network can be an Ethernet, a wireless access network, a wireless local area network (WLAN), etc.

[0122] ​The memory 901 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this.

[0123] As a possible implementation, the memory 901 can exist independently of the processor 902, and the memory 901 can be connected to the processor 902 through the bus 904 for storing instructions or program codes. When the processor 902 invokes and executes the instructions or program codes stored in the memory 901, the GPU communication method provided by the embodiments of the present disclosure can be implemented.

[0124] In another possible implementation, the memory 901 can also be integrated with the processor 902.

[0125] The bus 904 can be an extended industry standard architecture (EISA) bus or the like. The bus 904 can be divided into an address bus, a data bus, a control bus, and the like. For the sake of brevity and simplicity, Figure 9 Only one thick line is used in the figure to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0126] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the above division of functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the electronic device is divided into different functional modules to complete all or part of the functions described above.

[0127] In an example embodiment, the present disclosure also provides a readable storage medium including software instructions, which, when executed in an electronic device, can cause the electronic device to implement the method described in the above embodiments. The readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the readable storage medium can include both an internal storage unit of the electronic device and an external storage device. The readable storage medium is used to store the software instructions and other programs and data required by the electronic device. The readable storage medium can also be used to temporarily store data that has been output or will be output.

[0128] In an example embodiment, the present disclosure also provides a computer program product including computer instructions, which, when executed on an electronic device, cause the electronic device to perform the method in the above method embodiments.

[0129] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer-executable instructions. When the computer-executable instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer-executable instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or include one or more data storage devices that can be integrated with the medium, such as servers, data centers, etc. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), or an optical medium (such as a DVD), etc.

[0130] Although the present application has been described in connection with the embodiments thereof with reference to the various drawings, it will be understood that other variations and modifications of the present application will become apparent to those of ordinary skill in the art once they learn of the general inventive concepts thereof as reflected in the appended claims. The claims should not be limited to the embodiments set forth herein for the purpose of exemplification as such, but should be given a broad construction. In the claims, the use of "comprising" does not exclude other elements or steps not listed in the claims. The use of "a" or "an" does not exclude a plurality. A single processor or other unit can fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0131] Although the present application has been described in connection with the embodiments thereof with reference to the various drawings, it will be understood that other variations and modifications of the present application will become apparent to those of ordinary skill in the art once they learn of the general inventive concepts thereof as reflected in the appended claims. The claims should not be limited to the embodiments set forth herein for the purpose of exemplification as such, but should be given a broad construction. In the claims, the use of "comprising" does not exclude other elements or steps not listed in the claims. The use of "a" or "an" does not exclude a plurality. A single processor or other unit can fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0132] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A GPU communication method, characterized in that: The method is applied to a computing node and includes: Obtaining communication request information; the communication request information is used to request forwarding transmission data from the source GPU to the destination GPU; Determining the device types of the source GPU and the destination GPU based on the communication request information; Determining a target communication library based on the device types of the source GPU and the destination GPU; The transmission data of the source GPU is forwarded to the destination GPU through the target communication library.

2. The method according to claim 1, characterized in that The determining of a target communication library based on the device types of the source GPU and the destination GPU includes: When the device type of the source GPU and the device type of the destination GPU are both target device types, the target communication library corresponding to the target device type is determined based on the target device type and a preset first correspondence relationship; the first correspondence relationship is used to represent the correspondence between the device type and the communication library.

3. The method according to claim 1, characterized in that The communication request information includes an identifier of the source GPU and an identifier of the destination GPU; and determining the device types of the source GPU and the destination GPU based on the communication request information includes: Based on the identifier of the source GPU, the identifier of the destination GPU and a preset second correspondence, the device type of the source GPU and the device type of the destination GPU are determined; the second correspondence is used to represent the correspondence between the GPU identifier and the device type.

4. The method according to claim 2, characterized in that The computing node includes a plurality of homogeneous communication groups, wherein the GPUs in a homogeneous communication group have the same device type; the communication request information includes an identifier of the source GPU and an identifier of the destination GPU; and the method further includes: In response to the identifier of the source GPU and the identifier of the destination GPU being identifiers within a target homogeneous communication group, it is determined that the device type of the source GPU and the device type of the destination GPU are both target device types corresponding to the target homogeneous communication group.

5. The method according to claim 4, characterized in that The method further comprises: In response to the identifier of the source GPU or the identifier of the destination GPU satisfying at least one of the following, determining that the device type of the source GPU is different from the device type of the destination GPU: The identifier of the source GPU and the identifier of the destination GPU are identifiers of GPUs in different homogeneous communication groups respectively; The identifier of the source GPU is the identifier of the last GPU in any homogeneous communication group; The identifier of the destination GPU is the identifier of the first GPU in any homogeneous communication group.

6. The method according to claim 5, characterized in that The method further comprises: In a case where the device type of the source-end GPU is different from the device type of the destination-end GPU, the transmission data of the source-end GPU is forwarded to the destination-end GPU via the transit CPU.

7. The method according to claim 6, characterized in that The transit CPU includes a source CPU and a destination CPU; the source CPU is in communication with the source GPU; the destination GPU is in communication with the destination GPU; and forwarding the transmission data of the source GPU to the destination GPU through the transit CPU includes: Copying the transmission data of the source GPU to the remote direct memory access RDMA buffer of the source CPU; Calling an RDMA interface to transfer the transmission data from the RDMA buffer of the source CPU to the RDMA buffer of the destination CPU; The transmission data is copied from the RDMA buffer of the destination CPU to the destination GPU.

8. An electronic device, characterized in that: The electronic device includes: a plurality of graphics processing units (GPUs); the electronic device also includes: a memory and a processor; The memory stores instructions executable by the processor; When the processor is configured to execute the instructions, the electronic device implements the method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that include: Software instructions; When the software instructions are executed in an electronic device, the electronic device is enabled to implement the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that include: Computer instructions; When the computer instructions are executed in an electronic device, the electronic device is enabled to implement the method according to any one of claims 1 to 7.

Images